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Abstract 

This  dissertation  research  makes  significant  contributions  towards  the  synergistic  use  of  both 
analytical  and  simulation  models  for  improving  the  efficiency  of  simulation  studies.  The  founda¬ 
tion  for  this  research  is  the  application  of  the  analytical  control  variate  (ACV)  method.  The  ACV 
method  employs  an  external  analytical  model  to  consolidate  multiple  input  random  variables  into  a 
single  ACV.  Previous  research  suggests  that  this  approach  can  produce  significant  variance  reduc¬ 
tion,  but  the  resulting  point  estimate  of  the  simulation  response  may  exhibit  unacceptable  bias.  In 
this  research  a  general  Monte  Carlo  sampling  method  for  resolving  the  bias  problem  is  developed 
and  demonstrated  through  a  queueing  network  example.  In  order  to  use  the  method,  the  means, 
variances,  and  approximate  distributions  of  the  random  variables  used  to  produce  the  ACV  must 
be  known. 

For  some  simulation  models,  not  all  of  the  means,  variances,  and  approximate  distributions 
of  the  random  variables  used  to  produce  the  ACV  are  known.  In  this  research  both  parametric 
and  non-parametric  alternatives  to  the  Monte  Carlo  method  are  explored  for  these  cases.  The 
effectiveness  of  these  methods  is  demonstrated  using  an  airfield  simulation  model. 

Significant  variance  reduction  using  an  ACV  indicates  that  the  outputs  of  both  models  are 
highly  correlated  when  subjected  to  similar  inputs.  This  relationship  is  exploited  in  this  research 
and  a  new  methodology  is  developed  for  conducting  searches  of  a  simulation  design  space  using  an 
analytical  model  vice  a  simulation  model.  The  justification  for  the  new  surrogate  search  method 
is  based  on  validating  the  analytical  model  to  the  simulation  model  using  techniques  adapted  from 
simulation  model  validation  and  verification.  The  validation  and  surrogate  search  method  are  fully 
integrated  within  the  context  of  a  simulation  study  by  analyzing  the  results  of  the  ACV  method. 
The  effectiveness  of  the  method  is  demonstrated  on  two  simulation  models  including  the  HQ  AMC 
Mobility  Analysis  Support  System  (MASS)  model. 
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Efficient  Simulation  via  Validation 
and  Application  of  an 
External  Analytical  Model 


L  Introduction 

1,1  General  Discussion 

Air  Force  and  industry  analysts  use  mathematical  models  to  study  systems  and  provide 
decision-makers  with  the  information  necessary  to  set  policy  and  allocate  scarce  resources.  Exam¬ 
ples  of  mathematical  models  include  simulation,  statistical  regression,  and  stochastic  analytical. 
Each  of  these  different  types  of  models  has  qualities  that  recommend  or  discourage  their  use  for  a 
particular  system  or  problem.  All  three  of  these  types  of  mathematical  modelmg  are  discussed  and 
used  in  this  dissertation,  with  the  focus  on  simulation  and  stochastic  analytical  modeling. 

Large,  complex  systems  with  stochastic  elements  are  often  studied  with  discrete  event  simu¬ 
lation  models  rather  than  analytical  models.  The  major  advantage  of  using  a  simulation  model  is 
the  ability  to  model  system  characteristics  that  currently  defy  analytical  description.  The  major 
disadvantage  of  simulation  models  is  the  large  amount  of  time  required  to  complete  a  sufficiently 
accurate  study.  On  the  other  hand,  an  analytical  model  can  often  provide  a  solution  in  a  rela¬ 
tively  short  time  as  compared  to  a  simulation  study.  Unfortunately,  few  analytical  models  exist  for 
large,  complex  systems  since  such  systems  usually  have  characteristics  that  do  not  yield  to  analyt¬ 
ical  description.  On  the  other  hand,  if  the  system  can  be  solved  analytically,  the  complexity  of  the 
system  may  necessitate  solutions  that  are  equally  complex  with  imtenable  computational  or  numer¬ 
ical  problems.  Further,  because  of  the  complexity  even  approximate  analytical  models  are  seldom 
used  to  study  these  systems  despite  the  obvious  time  advantage  they  can  provide.  This  disserta- 
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tion  makes  significant  contributions  towards  the  S3mergistic  use  of  both  analytical  and  simulation 
models  to  reduce  the  time  required  to  complete  a  simulation  study. 

Many  techniques  and  methods  exist  for  reducing  the  time  necessary  to  complete  a  simulation 
study  including  those  referred  to  as  variance  reduction  techniques  (VRTs).  Due  to  the  nature  of 
discrete  event  simulation  models,  several  different  means  of  actually  reducing  the  observed  variance 
of  performance  measure  estimators  are  available  to  the  analyst.  By  reducing  the  variance  of  the 
estimator,  the  associated  confidence  interval  is  also  reduced.  Hence,  fewer  simulation  replications 
are  necessary  to  reach  a  pre-determined  level  of  accuracy,  reducing  the  time  of  a  study.  The 
method  of  control  variates  (CV)  is  a  VRT  that  takes  advantage  of  the  correlation  between  a  CV 
(a  random  variable)  and  the  simulation  output  estimator  (another  random  variable)  in  order  to 
achieve  variance  reduction.  Depending  on  the  type  of  CV  applied,  the  correlation  might  arise 
naturally  during  the  course  of  simulation  {internal  control  variates)  or  might  be  induced  by  using 
common  random  numbers  in  a  separate  control  simulation  model  {external  control  variates)  [18]. 

One  CV  method  that  combines  analytical  and  simulation  models  is  the  analytical  control 
variate  (ACV)  method  [49].  The  ACV  method  uses  a  separate  analytical  model  to  generate  a  CV 
that  is  correlated  to  the  output  of  a  simulation  model.  To  obtain  the  necessary  correlation,  the 
analytical  model  need  only  be  an  approximate  representation  of  either  the  same  tmderlying  system 
that  generated  the  simulation  model  or  the  simulation  model  itself.  Briefly,  the  ACV  method  is 
a  hybrid  of  the  two  types  of  typical  CV’s — ^internal  and  external.  The  moment  estimators  of  the 
realized  input  random  number  streams  that  drive  the  simulation  model  are  used  (like  internal  CV’s) 
as  inputs  to  an  external  (like  external  CV’s)  analytical  model  to  generate  the  ACV.  If  the  analytical 
model  is  an  adequate  representation  of  the  system  imder  study  (or  the  simulation  model),  the  ACV 
will  be  sufficiently  correlated  to  the  simulation  output  estimator  to  produce  adequate  variance 
reduction.  Unfortunately,  previous  researchers  [48,49,53,54]  have  all  reported  xmacceptable  levels 
of  bias  in  the  ACV  controlled  estimators.  This  bias  is  caused  by  the  necessity  to  evaluate  the 
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expected  value  of  the  ACV  given  the  distribution  of  the  input  random  variables  used  to  produce 
the  ACV  [49].  Since  the  analytical  models  developed  for  these  stochastic  systems  are  normally 
non-linear,  or  even  algorithmic  in  nature,  an  analytical  solution  of  this  model’s  expected  output 
is  extremely  difficult  if  not  impossible  to  generate.  We  present  an  efficient  solution  to  the  bias 
problem  in  this  dissertation  which  is  itself  a  significant  contribution  to  the  use  of  analytical  and 
simulation  models  in  concert  to  reduce  study  times. 

Now,  the  fact  that  the  anal3rtical  model  produces  variance  reduction  suggests  that  it  might 
be  possible  to  use  it  in  other  ways  to  reduce  simulation  study  times  since  given  similar  inputs  the 
two  models  produce  similar  outputs.  A  logical  next  step  is  to  exploit  this  relationship  and  use  the 
analytical  model  in  place  of  the  simulation  model  when  it  is  shown  to  be  valid  for  such  a  purpose. 
We  present  an  advancement  in  the  area  of  using  analytical  and  simulation  models  in  an  iterative 
fashion.  The  newly  developed  method,  called  the  surrogate  search  method,  assumes  that  the  ACV 
method  has  produced  significant  variance  reduction  so  that  the  analytical  model  can  be  used  as  a 
surrogate  for  the  simulation  model  to  perform  explorations  of  the  simulation  experimental  design 
space.  Using  the  anal3d;ical  model  instead  of  the  simulation  model  to  perform  this  function  can  save 
significant  amounts  of  time.  We  present  a  general  methodology  and  apply  it  to  a  study  involving 
a  large-scale  simulation  model. 

1.2  Problem  Statement 

The  sponsor  of  this  research,  Air  Mobility  Command  (AMC),  uses  two  discrete  event  simu¬ 
lation  models  to  analyze  their  operations — the  Base  Resource  and  Airfield  Capability  Evaluation 
(BRACE)  and  the  Mobility  Analysis  Support  System  (MASS)  model.  They  developed  BRACE 
as  an  airfield  simulation  used  to  estimate  an  airfield’s  throughput  capacity  and  resource  require¬ 
ments  [Ij.  BRACE  simulates  the  scheduled  flow  of  aircraft  to  an  airfield  with  predetermined 
resources.  All  the  major  groimd  activities  each  aircraft  must  accomplish  before  it  can  depart 
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the  airfield  are  also  simulated.  These  activities  include  taxiing,  scheduled  and  unscheduled  mainte¬ 
nance,  refueling,  cargo  upload  and/or  download,  and  passenger  movement.  The  resources  simulated 
comprise  a  runway,  ramp  parking  spots,  fuel  resources,  and  cargo  resources.  Fuel  trucks,  truck  refu¬ 
eling  stands,  fuel  hydrant  refueling  pits,  hydrant  laterals,  hydrant  fuel  tanks,  and  bulk  fuel  storage 
are  the  fuel  related  resources  simulated  in  BRACE.  The  cargo  resources  include  K-loaders,  forklifts, 
loading  docks,  and  warehouse  storage  facilities.  Among  the  many  possible  performance  measures 
that  BRACE  can  provide  for  AMC  decision  makers  are  aircraft,  cargo,  or  passenger  throughput, 
or  the  number  of  resources  required  to  meet  a  specific  throughput  [1]. 

MASS  simulates  the  AMC  global  airlift  system  and  is  capable  of  simulating  AMC  policies, 
procedures,  operations,  aircraft,  air  bases,  cargo,  passengers,  and  support  resources  as  they  relate 
to  the  airlift  system  [11].  MASS  simulates  a  fieet  of  aircraft  moving  a  given  amount  of  cargo  and 
passengers  from  any  number  of  on-load  points,  through  any  needed  en-route  stops,  to  any  number 
of  off-load  points,  then  recovering  and  returning  to  home  station  for  another  mission.  The  model 
can  continue  this  process  for  as  many  simulated  days  as  desired,  or  until  all  requirements  have  been 
airlifted  to  their  destination  [11]. 

Both  BRACE  and  MASS  are  large  simulation  models  and  significant  amounts  of  time  are 
required  to  accomplish  any  specific  study  using  them.  Reducing  the  time  it  takes  to  accomplish  a 
BRACE  or  MASS  study  is  of  much  interest  to  AMC.  We  demonstrate  in  this  dissertation  the  use  of 
an  analytical  model  in  concert  with  BRACE  and  MASS  to  reduce  the  time  of  studies  based  on  the 
methods  described  above.  In  order  to  accomplish  that  goal  the  ACV  bias  problem  is  resolved  and 
specific  methods  and  justifications  are  developed  for  the  surrogate  search  method.  These  issues  are 
briefly  discussed  in  the  next  section. 
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1.3  Dissertation  Issues 


1.3.1  ACV  Bias  Resolution  under  Known  Probability  Structures.  Previous  researchers 
have  failed  to  provide  a  solution  to  the  bias  observed  in  the  point  estimates  produced  when  em¬ 
ploying  the  ACV  method.  To  apply  the  ACV  method  to  a  real  simulation  model,  this  bias  problem 
had  to  be  resolved,  A  particular  resolution  to  this  problem  for  simulation  models  that  possess 
specific  properties  has  been  completed  and  is  presented  in  this  dissertation.  The  research  develops 
a  general  Monte  Carlo  method  of  distribution  sampling  that  resolves  the  bias  problem.  The  Monte 
Carlo  method  is  demonstrated  using  a  queueing  network  example.  This  Monte  Carlo  method  is 
applicable  if  the  mean  and  variance  of  the  input  random  variables  used  to  produce  the  ACV  are 
known.  Additionally  the  distribution  of  these  random  variables  must  be  known  either  exactly  or 
approximately.  This  is  a  simulation  model  specific  requirement.  In  many  cases,  this  requirement 
is  not  difficult  to  meet.  The  ACV  Monte  Carlo  method  is  demonstrated  to  perform  favorably 
when  compared  to  internal  and  external  CV’s.  Additionally  the  efficiency  of  the  ACV  Monte  Carlo 
method  is  demonstrated. 

1.3.2  ACV  Bias  Resolution  without  Complete  Probability  Knowledge.  For  some  simu¬ 
lation  models,  the  expected  value  of  the  moments  of  the  input  random  variables — ^the  inputs  to 
the  analytical  model — are  not  always  known.  For  example,  a  proportion  that  is  required  for  the 
anal3rtical  model  may  be  the  result  of  a  rule  within  the  simulation  model  instead  of  a  strict  random 
number  draw.  In  that  case  the  mean  and  variance  of  the  proportion  are  not  known  parameters  of 
a  random  variable  that  is  an  input  of  the  simulation  model.  Therefore,  the  Monte  Carlo  method 
described  above  cannot  be  used  in  those  situations. 

BRACE  presents  just  such  a  situation  when  it  simulates  aircraft  refueling.  BRACE  simulates 
aircraft  refueling  using  both  hydrant  systems  and  fuel  trucks.  The  mean  time  required  to  refuel  an 
aircraft  is  diflFerent  for  the  two  types  of  refueling  and  some  proportion  of  aircraft  will  be  refueled 
by  hydrant  while  the  other  aircraft  are  refueled  by  fuel  truck.  An  appropriate  analytical  model 
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that  accounts  for  this  diflFerence  in  mean  refueling  time  must  also  account  for  the  proportion  of 
aircraft  receiving  each  type  of  refueling.  In  BRACE,  this  proportion  is  based  on  a  simple  rule: 
when  aircraft  arrive  to  the  airfield  they  are  parked  at  a  ramp  spot  with  a  hydrant  system  refueling 
pit  if  ope  is  available.  Otherwise,  it  will  be  parked  at  a  spot  without  a  refueling  pit  and  receive  fuel 
from  a  fuel  truck  [1].  The  proportion  of  aircraft  that  will  be  refueled  at  a  fuel  pit  depends  on  many 
factors  including  aircraft  arrival  rates  and  the  time  aircraft  spend  on  the  airfield.  This  proportion 
can  only  be  known  as  a  result  of  performing  simulation  replications.  MASS  also  provides  numerous 
examples  of  analytical  model  inputs  that  have  unknown  distributional  parameters. 

Both  non-parametric  and  parametric  alternatives  to  the  Monte  Carlo  method  are  presented  in 
this  dissertation  using  a  simulation  model  based  on  BRACE.  To  conduct  this  research,  a  queueing 
network  analytical  model  employing  state-of-the-art  techniques  is  developed  to  produce  ACV’s  for 
the  BRACE  model.  We  apply  non-parametric  re-sampling  methods  such  as  the  bootstrap  [20]  and 
SIMDAT  [50]  that  do  not  require  any  knowledge  of  the  moments  or  probability  distributions  of  the 
analjdical  model  inputs  to  estimate  the  ACV  mean.  The  bootstrap  method  re-samples  realized  data 
points  (vectors)  while  the  SIMDAT  method  actually  generates  new  psuedo-data  points  based  on 
a  non-parametric  density  function  estimator.  The  re-sampled  points  from  both  methods  are  used 
to  approximate  the  mean  of  the  ACV  when  knowledge  of  the  probability  structure  is  unknown. 
A  parametric  approach  is  also  used  in  this  dissertation.  As  in  the  Monte  Carlo  method,  the 
central  limit  theorem  is  invoked  so  that  the  distributions  of  the  inputs  are  assumed  to  be  normally 
distributed.  The  difference  here  is  that  not  all  parameters  of  the  multivariate  normal  distribution 
are  known.  Different  schemes  of  estimating  all  or  some  of  those  parameters  are  explored.  Finally, 
combinations  of  these  methods  are  also  examined.  The  different  methods  are  compared  for  their 
later  use  in  this  dissertation. 

1.3.3  Surrogate  Search.  A  new  methodology  is  developed  in  order  to  justify  and  perform 
a  surrogate  search.  The  justification  for  using  an  analytical  model  in  place  of  a  simulation  model 
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is  based  on  classic  simulation  model  validation  and  verification  techniques.  The  validation  and 
verification  techniques  provide  a  framework  for  simulation  analysts  to  demonstrate  that  a  simulation 
model  is  a  valid  representation  of  the  system  under  study.  We  adapt  these  methods  for  our  own 
use  in  showing  that  the  analytical  models  we  employ  are  valid  representations  of  the  simulation 
model  imder  study.  The  surrogate  search  validation  and  verification  methods  rely  upon  the  results 
of  the  ACV  method  and  are  fully  integrated  into  the  steps  necessary  to  complete  a  simulation 
study.  Following  our  presentation  on  validating  the  anal3dical  model  as  a  surrogate,  we  derive  and 
describe  the  methods  necessary  to  perform  a  surrogate  search. 

To  demonstrate  the  effectiveness  of  the  surrogate  search  method,  aspects  of  a  response  surface 
methodology  (RSM)  study  are  made  on  a  simulation  model  based  on  BRACE  and  on  the  MASS 
simulation  model  itself.  Simply  put,  RSM  consists  of  several  statistical  techniques  for  empirical 
model  building.  The  methodology  describes  a  means  of  careful  design  and  analysis  of  experiments 
in  order  to  most  efficiently  relate  a  response  (or  output)  random  variable  to  the  levels  of  a  number  of 
predictor  (or  input)  random  variables  [12].  RSM  can  be  applied  to  any  number  of  systems  including 
discrete  event  simulation  models  and  is  an  excellent  vehicle  to  develop  the  surrogate  search  idea 
since  an  RSM  study  can  provide  for  many  opportunities  to  conduct  experimental  searches  within 
a  single  study.  The  use  of  an  analytical  model  in  conjunction  with  a  simulation  model  during  an 
RSM  study  can  result  in  significant  time  savings  since  the  number  of  simulation  replications  can 
be  significantly  reduced  via  variance  reduction  and  the  surrogate  search  methods 

The  RSM  study  conducted  on  the  model  based  on  BRACE  consists  of  a  simple  two-factor 
design  meant  to  illustrate  the  surrogate  search  method  and  highlight  basic  issues  of  the  method. 
On  the  other  hand,  the  RSM  study  conducted  using  MASS  consists  of  a  “real-world”  sized  problem 
using  an  actual  Air  Force  simulation  model.  Several  issues  are  addressed  in  this  demonstration. 
First,  an  analytical  model  of  MASS  is  constructed  in  order  to  complete  the  study,  the  first  time 
such  a  model  has  been  built.  Secondly,  two  new  performance  measures  for  determining  the  most 
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efficient  movement  of  cargo  are  developed  for  the  purposes  of  this  study.  Finally  resolutions  to 
several  non-standard  surrogate  search  issues  are  presented.  In  both  studies,  the  surrogate  search 
method  is  shown  to  achieve  the  goal  of  reducing  simulation  study  times. 

Overview 

This  dissertation  is  organized  in  the  following  manner.  Following  this  introduction  is  a  liter¬ 
ature  review  chapter.  The  areas  of  control  variates,  mean  value  analysis  queueing  network  models, 
and  response  surface  methodology  axe  each  addressed.  This  is  followed  by  a  short  chapter  on  the 
characteristics  of  simulation  and  analytical  models.  The  next  chapter  presents  the  research  com¬ 
pleted  on  resolving  the  ACV  method  bias  problem  with  known  distribution  parameters.  A  chapter 
foUows  this  on  resolving  the  bias  problem  without  complete  distributional  knowledge.  The  surro¬ 
gate  search  method  is  developed  in  the  next  chapter  followed  by  a  chapter  that  contains  the  two 
demonstrations  of  the  method.  The  first  demonstration  is  a  simple  RSM  study  conducted  on  a 
simulation  model  based  on  BRACE.  The  final  demonstration  is  a  real-world  sized  RSM  study  con¬ 
ducted  using  MASS.  The  fiinal  chapter  is  a  discussion  of  contributions  to  the  research  community 
as  a  result  of  the  research  completed  for  this  dissertation.  Included  will  be  recommendations  for 
future  research.  A  glossary  of  acronyms  and  abbreviations  that  appear  in  this  document  is  included 
for  the  reader’s  benefit  as  an  appendix. 
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IL  Literature  Review 


2.1  Overview 

The  following  discussion  presents  the  pertinent  literature  on  the  three  main  topic  areas  re¬ 
quired  to  complete  this  dissertation  research — control  variates  (CV’s),  analytical  modeling  using 
the  Mean  Value  Analysis  (MVA)  algorithm,  and  response  surface  methodology  (RSM).  The  control 
variate  section  contains  a  presentation  of  the  theory  of  control  variates,  control  variate  selection, 
and  the  main  categories  of  control  variates.  The  section  on  analytical  modeling  begins  by  describ¬ 
ing  the  types  of  multi-class  queueing  networks  that  have  exact  analytical  solutions  followed  by  a 
description  of  MVA  and  the  MVA  algorithm.  The  section  concludes  with  a  discussion  of  an  ap¬ 
proximate  MVA  algorithm  for  networks  with  fork-join  constructs.  This  type  of  analytical  network 
is  appropriate  for  one  of  the  example  studies  imder  consideration.  Finally,  the  last  section  in  this 
chapter  presents  an  overview  of  RSM. 

2.2  Control  Variates 

The  method  of  control  variates  is  an  effective  and  practical  variance  reduction  technique  for 
discrete  event  simulation.  Advantages  of  this  technique  include  low  computational  overhead  and 
applicability  to  a  wide  range  of  models. 

2.2.1  Control  Variate  Theory.  Several  textbooks  present  excellent  references  on  the  use 
of  control  variates.  The  Law  and  Kelton  textbook  [32]  on  simulation  modeling  is  one  example. 
These  authors  provide  an  overview  on  control  variate  theory,  application,  and  a  brief  discussion  on 
the  difference  between  internal  and  external  control  variates.  In  addition  to  their  excellent  coverage 
of  control  variates,  they  also  provide  an  extensive  reference  list  of  control  variate  literature. 

Nelson  [35]  presents  a  guide  for  simulation  practitioners  for  applying  three  variance  reduction 
techniques,  including  control  variates.  Nelson  provides  methods  for  finding  point  and  interval  esti¬ 
mators,  software  requirements,  and  guidelines  for  experimental  design  when  using  control  variates. 
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This  article  is  designed  as  a  tutorial,  not  as  a  definitive  presentation  of  control  variate  theory. 
As  such  it  is  very  useful  for  understanding  how  the  theory  of  control  variates  is  applied  to  actual 
simulation  studies.  The  following  discussion  summarizes  control  variate  theory  as  presented  in 
Nelson  [35]  and  Law  and  Kelton  [32]. 

Suppose  we  are  emplo3dng  simulation  to  estimate  E\Y]  where  the  random  variable  Y  is  the 
steady-state  waiting  time  for  customers  in  a  queueing  system.  To  reduce  variance  in  the  estimate  of 
/X,  we  might  exploit  another  random  variable  C  (e.g.  interarrival  time),  that  is  positively  correlated 
with  Y  and  has  known  expectation  —  E[C\.  A  new  random  variable  Y(b)  =  Y"  —  b(C  —  fi^)  can 
be  constructed  for  each  simulation  replication.  The  expectation 

E[Y{b)]  =  E[Y]  -  b{E[C]  -  fic)  (2.1) 

is  an  unbiased  estimator  of  /x  for  any  real  number  6.  Since 

Var{Y{b))  =  Var{Y)  +  b^Var{C)  -  2bCov{Y,  C)  (2.2) 


it  is  clear  that  Var{Y{b))  will  be  lower  than  Var{Y)  if  and  only  if 


2bC(yv{Y,  C)  >  b^Var{C)  (2.3) 

This  relationship  holds  if  Y  and  C  are  highly  correlated  and  b  is  selected  appropriately.  From 
Equation  (2.2),  the  value  of  b  that  minimizes  Var{Y{b))  is  given  by 


Cov{Y,C) 
^  Var{C) 


(2.4) 
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Normally  Cov{Y,  C)  is  not  known  but,  for  n  simulation  replications,  ^  can  be  estimated  by  the 
moment  estimator 


EUiYi-Y){Cj-c) 


(2.5) 


where  Y  and  C  are  the  respective  sample  means  of  the  n  observations  of  Y  and  C .  A  controlled 
estimate  of  /x  can  then  be  obtained  as 


Y0)  =  Y-HC-fXc)  (2-6) 

We  observe  that,  since  P  is  not  independent  of  Y  and  C,  we  cannot  assume  that  Y 0)  is  in  general 
unbiased. 

This  method  can  be  generalized  for  g  >  1  control  variates  C  =  with 

respective  known  means  ? M2?  •  *  •  ? 

Y{h)  =  Y^  h\C^  Me)  (2-7) 


where  each  b  =  (61 ,  ^2 ,  •  •  •  ?  bq)'  is  a  vector  of  real  numbers.  If  b  is  estimated  in  the  same  manner 
as  in  Equation  (2.5)  above,  and  Y  and  C  are  distributed  multivariate  normal,  Lavenburg  and 
Welch  [31]  have  shown  that 

Far  (y(  ;a))  =  -  R^r  c)Far(y)  (2.8) 

where  P  =  {0^ ,  /32, . . .  ,  ffqY  is  the  vector  of  optimal  estimates  of  b,  and  Ry  c  is  the  coefficient  of 
multiple  determination  between  Y  and  C.  This  coefficient  is  computed  as 

(2.9) 

try 
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where  cr^  is  the  variance  of  Y,  cry  c  and  or  cy  are  the  covariance  vectors  between  Y  and  C,  and 
Ec  is  the  covariance  matrix  of  C.  The  value  of  Ry  c  increases  as  each  additional  control  variate 
is  added,  reducing  variance.  However,  the  term  {n  —  2)/ {n  —  q  —  2)  increases  with  the  number  of 
control  variates.  Depending  on  which  effects  are  dominant,  the  addition  of  another  control  variate 
may  cause  the  variance  to  grow  rather  than  decrease  [35].  Several  methods  have  been  proposed  for 
effective  control  variate  selection  [2,9,39],  that  will  be  discussed  later. 

Lavenberg  and  Welch  [31]  offer  a  very  detailed  survey  on  the  application  of  control  vari¬ 
ates.  Other  areas  investigated  include  techniques  for  generating  control  variates  and  inefficiencies 
resulting  from  estimating  control  variate  coefficients.  Particularly  useful  are  the  two  appendices 
of  this  article  which  provide  detailed  discussions  on  the  application  of  control  variates  to  gener¬ 
ate  confidence  intervals.  The  first  appendix  describes  the  generation  of  confidence  intervals  using 
the  method  of  independent  replications  (and  equivalently  the  method  of  batch  means)  while  the 
second  appendix  presents  the  generation  of  confidence  intervals  using  the  regenerative  method  of 
simulation. 


Lavenberg  and  Welch  [31]  show  that  if  Y  and  C  have  a  joint  multivariate  normal  distribution, 
the  CV  estimator  is  unbiased.  Further,  control  variate  application  can  be  interpreted  as  a  classical 
regression  problem  given  the  following  joint  normality  assumption 
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Computationally,  this  is  a  simple  method  for  finding  Y{  "0)  and  estimating  it’s  variance  and  asso¬ 
ciated  confidence  interval.  Since 


E[Y\C=  c]  =  M+  /3'(c-//J 


(2.11) 
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the  regression  problem  can  be  stated  as 


Y=  X'r  +  e 


(2.12) 


where  Y  =  (Fi, I2,  •  •  •  j  Yn)',  e  is  the  prediction  error,  and 


with' 


1  cii  -  /iti  ...  c,i  -  Hg 


1  Citi 


^qn  Mg 


(2.13) 


(2.14) 


where  each  Cjj  is  the  ih  control  variate  of  the  jh  replication  (z  =  1, 2, . . .  ,g;  j  =  1,2, . . .  ,n).  To 


find  the  least  squares  estimators  set 


7  = 


A 


=  (  X'X)-^  X'Y 


(2.15) 


resulting  in  the  equation 


Y=  X 


A 


+  e 


(2.16) 
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Control  variates  can  also  be  formulated  for  multivariate  responses.  Several  papers  present 
methods  for  using  control  variates  for  multi-response  simulation  models.  See  for  example  [9,38,39, 
44,55,57].  The  extension  to  multivariate  response  is  straightforward  with  the  univariate  statistics 
replaced  by  their  multivariate  analogs. 
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Further,  not  all  control  variates  are  linear.  Nelson  [34]  describes  several  diflferent  forms  of 


control  variate  construction.  Other  possible 


control  variance  estimators  include  the  ratio  CV, 


the  polynomial  CV, 


^  ratio 


(2.22) 


=  Y-I3,{C-  fic)  -  iJicf  +  ■■■  (2-23) 


an  extension  of  the  ratio  CV  that  includes  a  multiplier 

Ymuim=Y[!^f^^ 


(2.24) 


and  the  power  CV 

•‘■power  — 


(2.25) 


2.2.2  Control  Variate  Bias.  Lavenberg  and  Welch  [31]  proved  that  CV  estimators  are 
unbiased  if  Y  and  C  have  a  joint  multivariate  normal  distribution.  However,  if  Y  and  C  are  not 
jointly  normal,  CV  estimators  are,  in  general,  biased  if  /3  is  estimated.  Nelson  [36]  proved  that 
regardless  of  the  distribution  of  {Y,  C)  the  following  central  limit  theorem  for  control  variates  holds 


n 


1/2 


[n3) 


N  [O,  CTy  (1  —  iiy  q)]  as  n  — >  00 


(2.26) 


Thus,  even  when  the  normality  assumption  is  not  appropriate,  the  asymptotic  property  2.26  justifies 
the  use  of  CV’s  as  long  as  n  is  sufficiently  large. 
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However,  if  n  is  not  very  large  and  the  normality  assumption  doesn’t  hold,  researchers  have 
suggested  several  remedies.  Tew  and  Wilson  [51]  present  a  method  for  checking  the  multivariate 
normality  assumption.  Nelson  [36]  presents  several  different  methods  to  combat  the  bias  problem 
including  splitting,  jackknifing,  and  bootstrapping.  He  makes  several  recommendations  on  the 
most  appropriate  method  based  on  the  number  of  replications.  Avramidis  and  Wilson  [3]  describe 
a  different  splitting  scheme  that  guarantees  imbiased  CV  estimation  without  any  distributional 
assumptions  at  the  price  of  a  “slight”  increase  in  confidence  interval  width. 

2.2,3  Control  Variate  Selection.  Selection  of  the  best,  or  nearly  the  best,  subset  of 
possible  control  variates  is  another  area  of  extensive  research.  Nelson  [35]  describes  two  methods 
for  deciding  which  subset  of  possible  control  variates  to  use  for  reducing  variance.  One  method  is 
to  use  a  regression  software  package  and  perform  stepwise  regression  on  all  possible  control  variates 
and  then  select  the  subset  of  controls  that  create  the  largest  amount  of  variance  reduction.  He  also 
proposes  a  means  of  determining  a  marginal  improvement  ratio  for  adding  an  additional  control 
variate  to  the  set  of  controls  already  in  use.  For  example,  for  a  set  of  q  control  variates,  the  marginal 
improvement  ratio  is  computed  by  comparing  1  —  R^  for  the  set  of  q  +  l  control  variates,  and  1  — 
for  q  control  variates,  where  is  an  estimate  of  the  square  of  the  multiple  correlation  coefficient 
of  the  response  variable.  He  provides  a  table  of  marginal  improvement  ratios  necessary  for  adding 
an  additional  control  variate  based  on  the  number  of  replications  performed  and  number  of  control 
variates  already  included. 

Bauer  and  Wilson  [9]  develop  a  method  for  selecting  the  best  subset  of  possible  control  vari¬ 
ates  for  multi-response  simulation  models.  Their  criteria  for  selection  minimizes  the  mean-square 
confidence  region  volume  for  responses  under  the  assumption  that  (  Y,  C)  is  distributed  multivari¬ 
ate  normal.  Porta  Nova  and  Wilson  [39]  develop  control  variate  selection  criteria  when  estimating 
multi-response  simulation  metamodels,  when  (  Y,  C)  is  not  jointly  normal.  They  consider  spe- 
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cific  covariance  structures  for  the  responses  and  possible  controls  that  occur  for  specific  types  of 
psychometric  and  econometric  simulation  studies. 

Internal  and  External  Control  Variates,  Existing  control  variate  methods  are  gener¬ 
ally  classified  as  either  internal  or  external  Internal  control  variates  may  be  input  random  variables 
or  arbitrary  functions  of  these  inputs,  and  deliberative  selection  of  the  right  combination  of  can¬ 
didate  variables  to  produce  a  low-variance  unbiased  estimate  of  the  response  can  be  a  complex 
undertaking.  Alternatively,  the  external  method  requires  the  creation  of  an  analytical  model  of 
a  simplified  version  of  the  system  under  study.  A  simulation  of  the  simplified  system  is  then  im¬ 
plemented  using  the  same  random  number  streams  as  the  original  simulation.  External  control 
variates  are  rarely  used  in  practice,  due  to  the  difficulty  in  obtaining  the  necessary  synchronization 
of  the  random  variate  generators  [32]. 

Examples  of  external  control  variates  can  be  found  in  Burt,  Gaver,  and  Perlas  [14]  and  Gaver 
and  Shedler  [23].  In  [14],  the  effectiveness  of  several  variance  reduction  techniques  on  project 
graph  analysis  (PERT,  GERT,  CPM,  etc.)  network  simulations  is  examined.  Among  the  different 
techniques  are  external  control  variates.  The  authors  generate  the  external  control  variates  by  first 
constructing  simplified  networks  that  have  analytical  solutions  and  are  “similar”  to  the  networks 
under  study.  They  then  simulate  both  models  separately  using  a  common  random  number  stream 
to  drive  both  simulations.  Gaver  and  Shedler  [23]  apply  external  control  variates  to  simulations  of  a 
multiprogrammed  computer  system.  In  a  similar  manner  as  in  the  previous  article,  they  propose  a 
model  of  the  system  and  a  similar  model  that  can  be  solved  analytically.  In  both  articles  significant 
reduction  in  variance  is  reported. 

Many  researchers  have  proposed  many  forms  of  internal  control  variates.  We  will  focus  on 
those  CV’s  that  will  be  considered  in  this  dissertation.  Standardized  work  variables  are  devel¬ 
oped  for  queueing  systems  with  the  regenerative  property  in  Wilson  and  Pritsker  [56].  Given  a 
service  process  Uj{k)  :  j  >  1,  at  service  center  k  with  known  expected  value  and  variance 
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standardized  work  variables  are  defined  as 


a(fe,t) 

Cu{t)  =  [a(fe,  Y.  (2.27) 

i=i 

where  a(A;,  t)  is  the  number  of  service  times  started  at  center  k  during  the  time  period  [0,  t].  These 
standardized  variables  were  developed  since  previous  internal  control  variates  [30]  have  been  shown 
to  have  an  asymptotic  variance  equal  to  zero.  This  property  causes  the  variance  covariance  matrix 
for  a  set  of  these  controls  to  be  asymptotically  singular.  Since  an  inverse  of  this  matrix  (or  usually 
an  estimate  of  it)  must  be  computed  to  find  (or  estimate)  the  optimal  control  coeflS-cient,  numerical 
problems  can  arise  as  replication  length  increases.  The  standardized  work  variables,  on  the  other 
hand,  have  an  asymptotic  variance  of  1.  Further,  Wilson  and  Pritsker  prove  that  standardized 
control  variables  for  queues  with  the  regenerative  property  converge  in  distribution  to  multivariate 
normal  with  a  mean  vector  of  0.  Experiments  on  a  simple  network  were  performed  with  substantial 
variance  reduction  reported. 

Another  set  of  standardized  internal  control  variables,  standardized  routing  variables,  are 
developed  in  Bauer  and  Wilson  [10].  These  can  be  used  for  discrete-event  simidation  models  that 
have  a  multinomial  construct.  Standardized  routing  variables  attempt  to  exploit  the  correlation 
between  departures  fi:om  the  mean  branching  probabilities  in  a  network  and  the  resultant  network 
response.  They  are  defined  in  the  following  manner.  Consider  a  multinomial  branching  process  of 
g  branches  leading  to  g  service  centers,  and  define  an  indicator  variable  as 


iiU)  =  { 


1  if  the  i~th  departing  customer  goes  to  center  j, 
0  otherwise 


(2.28) 
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Then  a  standardized  routing  variable  for  center  j  is  defined  as 


N(t) 


Ii{3)  -P{j) 


j  =  1,2,...  ,g 


(2.29) 


where  N(t)  is  the  total  number  of  transits  through  all  g  branches  in  the  time  interval  [0,  t]  and  p{j) 
is  the  probability  that  a  customer  is  sent  to  center  j,  Bauer  and  Wilson  performed  several  experi¬ 
ments  on  a  simple  network  with  results  indicating  significant  confidence  interval  length  reduction, 
particul^ly  when  the  standardized  routing  variables  are  used  in  conjunction  with  standardized 
work  variables. 


2.2.5  Analytical  Control  Variates.  For  many  simulation  studies,  it  may  be  possible  to 
avoid  the  technical  challenges  of  external  or  multiple  internal  control  variates  by  using  an  external, 
analytical  model  to  generate  a  single  “analytical”  control  variate  (ACV)  for  each  replication  of  a 
simulation.  Nelson  [34]  first  presented  the  idea  of  an  analytic  control  variate,  but  did  not  report 
any  experimental  results.  Sharon  [48]  and  Sharon  and  Nelson  [49]  used  Jackson  network  results 
to  generate  anal3rtical  control  variates  for  queueing  network  simulations.  Tomick  [53]  and  Tomick, 
Litko,  and  Bauer  [54]  demonstrated  that  the  approach  can  produce  significant  variance  reduction 
in  a  broader  range  of  queueing  network  models.  AH  of  these  researchers  also  reported  unacceptable 
levels  of  bias  in  the  response  estimate. 

Sharon  and  Nelson  [49]  describe  the  construction  of  an  ACV  by  first  describing  the  construc¬ 
tion  of  an  external  control  variate  (ECV).  As  described  above,  ECys  require  a  second  system  that 
is  similar,  yet  different  from  the  simulation  system  of  interest.  In  fact,  the  second  system  has  a 
known  analytical  or  numerical  approximation  of  where  ^  is  an  output  performance  measure  of 
the  second  system  that  corresponds  to  6,  the  output  performance  measure  of  interest  of  the  primary 
simulation  system.  Calling  the  second  system  the  control  system,  both  systems  are  simulated  using 
common  random  numbers.  If  C,  the  estimator  of  (j>  from  the  control  system,  is  strongly  correlated 
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to  y,  the  estimator  of  6  for  the  system  of  interest,  variance  reduction  will  occur.  Since  ECVs 
require  a  second  simulation,  significant  levels  of  variance  reduction  must  occur  for  ECVs  to  be  an 
eflBcient  means  of  variance  reduction. 

Nelson  [35]  su^ested  another  method  of  generating  an  external  control  variate.  In  this  case, 
let  5  be  a  vector  of  parameters  for  the  input  distributions  for  the  control  system  sudi  that  (f)  =  g{6), 
where  p  is  a  function  and  <f>  is  the  same  as  described  above.  If  the  system  of  interest  has  the  same 
input  parameters,  S,  but  is  different  from  the  control  system  in  other  ways,  the  new  method  can 
be  applied.  The  authors  point  out  that  the  key  step  is  to  simulate  the  system  of  interest  to 
obtain  Y  and  6,  where  5  is  an  estimator  of  the  known  quantity  S.  The  analytical  control  variate  is 
then  formed  hyY{b)  =  Y-b  (^g{S)  -  g{6)j .  For  K  replications  (y*, ,  5*,),  fe  =  1, 2, . . .  ,  A"  an  ACV 
controlled  estimator  is 

Y{b*)  =  Y-b*(g{6)-(j>)  (2.30) 

where  S*  estimates  b*  =  Cov  |^y,  /Var  .  The  authors  point  out  that  g(S)  may  not  be  an 
unbiased  estimator  of  (f). 

One  advantage  of  an  ACV  to  an  ECV,  according  to  the  authors,  is  that  an  ACV  doesn’t 
require  the  extra  time  required  to  generate  a  second  simulation.  Also,  the  problems  associated 
with  applying  common  random  numbers  are  not  encountered.  All  that  is  required  is  to  apply 
ACVs  are  a  control  system,  the  function  g  (which  may  be  a  numerical  approximation  or  closed 
form  function),  and  the  vector  of  realized  input  parameters 

The  authors  perform  experiments  on  simple  queueing  networks  to  explore  the  effects  of  an 
ACV.  Their  results  indicate  significant  variance  reduction  under  most  of  the  experimental  settings. 
However,  they  do  experience  bias  in  results.  They  point  out  that  bias  of  the  controlled  estimator 
can  arise  from  the  estimation  of  b*  and  since  E  fp(5)l  ^  The  authors  point  out  that  these 
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are  important  effects  that  must  be  investigated  since  variance  reduction  at  the  expense  of  increased 
bias  may  not  be  acceptable  [49] . 

2.3  Analytical  Modeling 

When  constructing  an  analytical  model  for  the  purpose  of  generating  an  ACV,  the  model  need 
only  be  a  reasonably  accurate  representation  of  the  modeled  system.  Essentially,  the  model  should 
be  a  function  that  uses  some  subset  of  the  same  input  random  variables,  or  their  moments,  as  the 
simulation  model  and  has  as  its  output  the  same  measures  of  interest.  Additionally,  the  analytical 
model  should  be  easy  to  implement  and  reasonably  fast.  To  meet  these  requirements,  the  ACV 
analytical  model  considered  for  this  dissertation  will  be  a  closed  multi-chain  queueing  network.  By 
using  a  Mean  Value  Analysis  (MVA)  algorithm,  the  steady  state  expected  values  of  the  measures 
of  interest  can  be  found  rapidly  and  exactly  [13, 16, 29].  Such  an  approach  is  demonstrated  in  this 
dissertation  to  achieve  significant  levels  of  variance  reduction  at  a  low  computing  cost  when  applied 
to  a  small  simulation  model  of  a  multi-programmed  computer  system.  The  following  sections 
present  discussions  on  product  form  networks,  the  MVA  algorithm,  and  a  fork-join  queueing  network 
model. 

2.3.1  Product  Form  Networks.  The  basic  type  of  product  form  queuing  networks  are 
known  as  BCMP  networks,  named  for  Baskett,  Chandy,  Muntz,  and  Palacios.  These  authors 
describe  a  general  multi-chain  queueing  network  and  demonstrate  that  it  has  a  product  form  solu¬ 
tion  [8] .  At  the  time  of  their  publication,  BCMP  networks  described  the  most  general  network  with 
a  known  product  form  solution.  Although  several  authors  have  since  extended  their  work  in  very 
specific  areas  (see  [26]  for  example),  BCMP  networks  stiU  form  the  basic  guidelines  for  product 
form  networks.  For  that  reason,  we  wiU  first  concern  ourselves  with  networks  that  meet  the  BCMP 
requirements. 
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BCMP  networks  are  open,  closed,  or  mixed  networks  of  queues  or  service  stations  with  one 
or  more  classes  of  customers  that  have  one  of  the  four  following  service  disciplines: 

First  Come  First  Served  (FCFS)  Customers  are  served  in  the  order  of  their  arrival  by  a  single 
server.  The  service  time  for  all  customers  is  exponential  with  the  same  mean  for  all  classes 
of  customers.  Load  dependent  service  rates  are  allowed. 

Processor  Sharing  (PS)  Customers  are  serviced  by  a  single  server  with  a  processor  sharing 
(time  division)  service  discipline.  The  service  time  distribution  can  be  any  distinct  Coxian 
distribution  for  each  class  of  customer.  Load  dependent  service  rates  are  allowed. 

Delay  Station  (D)  This  service  station  has  an  infinite  number  of  servers  (or  at  least  as  many  as 
the  total  number  of  customers  in  the  network).  Each  class  of  customer  may  have  a  distinct 
Coxian  service  time  distribution. 

Last  Come  First  Served  (LCFS)  There  is  a  single  server  and  the  service  disciphne  is  last  come 
first  served  where  the  last  arriving  customer  has  an  absolute  priority  (the  currently  serviced 
customer  is  immediately  pre-empted).  Each  class  of  customer  may  have  a  distinct  Coxian 
service  time  distribution  with  load  dependent  service  rates. 

Each  class  of  customer  in  the  network  may  have  its  own  probabilistic  routing  chain  and  mul¬ 
tiple  servers  can  be  represented  by  using  load  dependent  service  rates.  Based  on  these  assumptions, 
Basket,  et.  al.  [8]  provide  product  form  formulas  for  the  equilibrium  state  probabilities.  They  prove 
that  the  formulas  are  correct  by  showing  that  they  satisfy  the  independent  balance  equations.  Once 
the  state  probabilities  are  foimd,  the  different  network  performance  measures,  such  as  mean  queue 
size,  mean  waiting  times,  and  throughput,  can  be  determined. 

2.3.2  Mean  Value  Analysis.  Mean  value  analysis  can  provide  the  same  performance 
measures  for  closed  or  capacitated  BCMP  networks  without  the  need  to  solve  for  the  equilibrium 
state  probabilities  [29,42].  In  particular,  MVA  can  calculate  mean  response  time  Ri  (waiting  and 
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service  time  at  service  center  z),  throughput  A^,  queue  length  Qi  (number  of  customers  waiting  and 
in  service  at  station  z),  and  server  utilization  Ui  (expected  number  of  busy  servers  at  station  z). 
Detailed  explanations  of  the  MVA  algorithm  can  be  found  in  Bruell  and  Balboa  [13]  and  in  Conway 
and  Georganas  [16]. 

The  foundation  of  MVA  is  the  arrival  theorem  first  proven  by  Lavenberg  and  Reiser  [29].  The 
theorem  states  that  for  a  closed  network  with  N  customers,  an  arriving  customer  to  service  center 
z  observes  the  same  distribution  of  customers  at  that  station  as  the  stationary  (random  observer’s) 
distribution  for  the  same  network  with  N  —  1  customers.  Consider  a  single  class  network  with  m 
service  stations  where  the  service  rate  at  station  z  when  n  customers  are  present  is  given  by  /z^(n). 
Then  the  arrival  theorem  can  be  used  to  develop  the  marginal  local  balance  theorem: 

l,,{n)Pi{n\N)  ==  Xi{N)Piin  -  l|iV  -*  1)  (2.31) 

where  Pi(n\N)  is  the  probability  that  n  customers  are  at  station  z  given  that  N  customers  axe 
in  the  network,  and  Xi{N)  is  the  customer  throughput  at  station  z  when  N  customers  are  in  the 
network.  By  applying  the  marginal  balance  equation  recursively,  the  performance  measures  can  be 
computed.  Mean  queue  length  can  be  found  by 

Qi(N)  =  “  1)  (2.32) 

n=l  n=l  ' 

The  value  of  throughput  Xi{N)  in  Equation  (2.32)  is  xmknown,  but  by  applying  Little’s  Law  we 
find  that 


(2.33) 
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If  station  i  has  only  one  server,  then 


Ri{N)  =  Si  53 ~  +  Qi{N-  1)]  (2.34) 

n—1 

where  Si  is  the  mean  service  rate  of  a  single  server  at  station  i.  If  station  i  is  an  mfimte  server 
station,  then  Ri{N)  =  s*  for  all  iV. 

Both  Equations  (2.33)  and  (2.34)  relate  the  response  time  of  any  station  when  N  customers 
are  in  the  system  to  the  distribution  of  customers  at  the  same  station  when  N  -1  customers 
are  in  the  network.  Thus,  by  beginning  calculations  when  iV  =  1  (so  that  Pi  (0|iV  —  1)  =  1 
and  Qi{N  -  1)  =  0  Vi),  all  performance  measures  can  be  found  recursively.  To  find  the  station 
throughputs  for  each  iteration,  the  av^age  cycle  time  for  a  customer  at  an  arbitrary  reference 
station  must  be  calculated.  If  we  use  station  1  as  the  reference  station,  the  average  time  between 
departures  firom  the  reference  station  for  the  same  customer  is  given  by 

i=l 

where  each  ratio  Vi/vi  is  the  mean  number  of  visits  a  customer  makes  to  station  i  for  every  visit 
to  station  1.  Then  station  throughput  can  be  solved  for  by 


Ai(JV)  = 


Nvi 

CTi{N)vi 


(2.36) 


Using  Little’s  Law,  the  mean  queue  length  and  station  utilization  can  be  found  by 


QiiN)  =  MN)Xi{N) 


(2.37) 


Ui{N)  =  sMN) 


(2.38) 
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If  station  i  is  a  single  server  station,  the  results  from  Elquation  (2.37)  can  be  applied  to  Equation 
(2.34)  to  calculate  the  mean  response  times  with  one  more  customer  in  the  system.  However,  if 
station  i  has  more  than  1  server,  the  marginal  local  balance  theorem  can  be  used  to  determine  the 
new  distribution  of  customers  as 


Pi(n|Ar) 


Ai(n)Pi(n-l|iV-l) 


n  >  0 


(2.39) 


N 

Pi{0\N)  =  1  -  X) (2-40) 

n=l 

These  probabilities  can  be  applied  to  Equation  (2.33)  to  calculate  the  multiple  server  response 
times  for  the  next  iteration.  This  process  is  repeated  until  N  equals  the  total  number  of  customers 
in  the  network.  These  formulas  can  be  extended  to  account  for  multi-class  networks.  Appropriate 
algorithms  can  be  found  in  Bruell  and  Balboa  [13]  and  Conway  and  Georganas  [16]. 

2.3.3  Fork- Join  Queueing  Network  Approximation.  One  important  construct  of  BRACE 
that  should  be  included  in  an  analytical  model  for  ACV  use  is  that  of  concurrent  service  activities. 
These  types  of  activities  cannot  be  modeled  as  a  product  form  network,  so  the  MVA  algorithm 
cannot  be  directly  applied  [29].  However,  a  fork-join  approximation  method  for  MVA  has  been 
developed  by  Dietz  and  Jenkins  [19]  from  the  work  of  Rao  and  Suri  [41].  The  fork-join  approxima¬ 
tion  allows  for  the  modeling  of  concurrent  activities  within  the  network.  A  fork-join  node  can  be 
described  in  the  following  manner.  A  customer  arriving  at  a  fork-join  node  generates  clones  that 
enter  the  separate  substations  of  the  node  and  are  rejoined  to  the  parent  customer  once  service  is 
completed  at  their  separate  substations.  Once  all  clones  complete  their  servicing,  the  parent  cus¬ 
tomer  is  made  whole  and  can  move  on  to  the  next  network  service  station  according  to  its  routing 
chain. 
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Dietz  and  Jenkins  [19]  derive  a  method  within  the  MVA  framework  to  approximate  network 
measures  for  multiple  fork-join  nodes  with  multiple  server  activities  and  probabilistic  service  re¬ 
quirements.  Their  method  can  be  summarized  in  the  following  manner.  Consider  a  fork-join  node  i 
that  contains  Ki  substations  where  the  probability  that  a  clone  proceeds  to  substation  k  of  station 
i  is  =  1  for  all  fe  =  1, . . .  ,  ifi.  Approximate  mean  response  time,  queue  length,  and  utilization 
for  clones  for  substations  ik  are  represented  by  Qik{N),  nd  Uik{N),  Two  approximations 

are  made  in  order  to  evaluate  network  performance  measures: 

•  Approximation  L  For  a  network  with  N  customers,  a  clone  arriving  at  a  substation  sees  the 
stationary  distribution  of  clones  at  the  substation  for  the  same  network  with  N—l  customers. 

•  Approximation  2.  The  response  time  experienced  by  a  clone  at  a  substation  can  be  represented 
as  a  exponentially  distributed  random  variable  and  is  independent  of  the  response  time  at 
other  substations. 

Using  Approximation  2,  let  the  response  time  for  any  substation  ik  be  denoted  by  the  ex¬ 
ponential  random  variable  Tik{N)  with  rate  parameter  6ik{N)  =  l/Rik{N),  This  response  time 
represents  both  substation  service  time  and  any  waiting  time  in  the  substation.  The  mean  time 
that  a  parent  customer  spends  at  fork-join  node  i  is  given  by  E[mBy:k=i,...  ,Ki  This  time 

is  then  used  to  calculate  network  cycle  time. 

If  we  let  S  represent  the  subset  of  all  possible  substations  that  any  particular  customer  will 
require  at  fork-join  node  i,  then  the  assumption  that  gi/c  =  1  for  all  fc  =  1, . . .  ,  Xi  can  be  relaxed 
by  conditioning  on  S,  Let  fli  be  the  union  of  all  possible  subsets  for  a  particular  fork-join  node  i, 
and  let  'Jri{S)  be  the  probability  that  subset  S  is  required  by  a  customer.  Note  that  the  number 
of  subsets  in  fli  is  given  by  ~  Assuming  that  the  selection  of  substations  is 

independent,  the  probability  that  a  customer  requires  any  particular  subset  S  at  fork-join  node  i 
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is  given  by 


MS)  =  llqikl[ii-qik)  (2.41) 

keS  k^S 

The  mean  of  the  conditional  holding  time  at  fork-join  node  i  is  E[msix.kes  (defined  as 

zero  if  5  =  0). 

Finding  the  conditional  mean  holding  time  is  mathematically  the  same  as  determining  the 
mean  time  to  failure  for  a  parallel  system  of  independent  components  with  exponentially  distributed 
failure  times  [25].  By  the  independence  described  in  Approximation  2,  the  cumulative  distribution 
function  (CDF)  for  conditional  holding  time  is 


m  =  '[[p{TikiN)<t}, 

k^S 

=  (2.42) 

kes 


Since  E[X]  =  —  F(i))dt  for  any  noimegative  continuous  random  variable  X  with  CDF  F{t), 


then 


(l-exp{-Oik{N)t}) 


dt 


f  1 1  -  1  +  V)  exp  {N)t} 

Jo  I  tts 

-  ^  ^exp  {-  {Sik  (N)  +  dii  (N))  t} 

k€S  l^S 
Kk 

exp  {-  idik{N)  +  diliN)  +  eirr,{N))t} 

k€Sl^S  m^S 

l<k  m<kfl 

-■■■  +  (-1)^(5)+!  exp |-  i|| di  (2.43) 
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where  K{S)  is  the  number  of  substations  in  S.  Evaluating  the  integral  yields 


£toi«{r„(jv))i 


E 

k€S 


1 

OikiN) 


-EE 

k€SleS 

Kk 


1 

Oik{N)  +0ii(N) 


+EEE 

k£Sl£S  meS 

Kk  Tn<k,l 


1 

eik{N)  +  eii{N)  +  eim{N) 


1 

^keS 


(2.44) 


The  MVA  algorithm  can  be  adjusted  for  a  network  with  a  set  /  of  simple  service  stations 
and  a  set  J  of  fork-join  nodes.  Response  times  for  simple  service  stations  are  computed  at  each 
iteration  using  Equations  (2.33)  and  (2.34).  For  the  fork-join  substations  with  multiple  servers, 
Approximation  1  allows  for  application  of  the  marginal  local  balance  theorem  and  Little’s  Law  so 
that 


For  single-server  fork-join  substations,  Approximation  1  leads  to 


Rik{N)  =  Sik  {1  -f  Qik{N  -  1)}  (2.46) 

Then  the  substation  response  rates  Oik{N)  =  l/iJjfc(iV)  can  be  used  to  find  the  mean  conditional 
holding  times  for  the  fork-join  nodes.  Then  cycle  time  for  station  1  is  given  by 


CTi{N)  = 


E 


Vl 


(2.47) 


Now  that  cycle  time  has  been  computed,  Equations  (2.36)  -  (2.38)  can  be  used  as  before  to  compute 
throughputs,  queue  lengths,  and  utilization  for  the  simple  service  stations.  Since  each  fork-join 
substation  ifc  is  visited  by  a  clone  an  average  of  qik  times  every  time  a  customer  visits  node  i,  the 
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substation  throughput  is  calculated  by 


XikiN)  = 


Nvjqik 

CTiiN)vi 


The  other  substation  performance  measures  are  found  using  Little’s  Law  by 


(2.48) 


Qik{N)  =  Ilik{N)Xik(N)  (2.49) 

Uik{N)  =  SikXikiN)  (2.50) 

An  iteration  of  the  algorithm  can  be  started  with  the  use  of  Equation  (2.33)  or  (2.34)  for  a 
simple  service  station  or  liquation  (2.46)  for  fork-join  nodes  with  single  servers.  For  fork-join  nodes 
with  multiple  server  substations,  Approximation  1  provides  clone  probability  distributions 

Pik{n\N)  =  „  >  0  (2.51) 

Mifc(«) 

N 

Pik{0\N)  -  1  -  X!  Afc(n|JV)  (2.52) 

n— 1 

These  probabilities  can  then  be  used  in  Equation  (2.45)  to  calculate  new  response  times.  Iterations 
are  continued  imtil  N  is  the  desired  number  of  customers  in  the  network. 

Dietz  and  Jenkins  [19]  tested  their  fork-join  approximation  on  an  aircraft  sortie  generation 
model.  Their  method  provided  highly  accurate  estimates  of  the  mean  performance  measures  for  the 
networks  studied  in  just  a  few  seconds.  The  results  were  most  aecurate  when  resource  utilization 
was  relatively  low. 
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Dietz  extended  the  fork-join  heuristic  approach  to  include  the  possibility  of  multiple  stations 
on  a  fork-join  path,  including  nested  fork-join  constructs  [17].  This  is  accomplished  by  changing 
approximation  2  to  read: 

•  Approximation  2.  The  transit  time  of  a  clone  along  a  fork-join  path  can  be  represented  as  an 
exponentially  distributed  random  variable  and  is  independent  of  the  transit  time  for  clones 
on  other  paths. 

As  before,  the  response  time  for  any  fork-join  substation  is  solved  as  before.  For  fork-join 
paths  with  more  than  one  substation,  or  a  nested  fork-join  path,  the  transit  time  along  the  path 
is  the  sum  of  all  sub-station  response  times  and  fork-join  transit  times  on  that  path,  where  each 
fork-join  path  transit  time  is  represented  by  an  exponentially  distribute  random  variable  [17].  Dietz 
demonstrates  the  accuracy  of  the  MVA  heuristic  by  comparing  results  of  the  heuristic  to  that  of  a 
simulation  model  of  military  airlift  field.  His  results  indicate  no  relative  error  exceeding  13%,  with 
most  errors  being  much  smaller  for  the  parameters  investigated. 

2,4  Response  Surface  Methodology 

Response  surface  methodology  (RSM)  is  a  set  of  statistical  and  mathematical  techniques 
for  empirical  model  building.  The  underlying  mechanisms  of  some  phenomena,  or  system,  are 
understood  well  enough  that  mathematical  models  that  are  the  result  of  this  understanding  can  be 
derived.  Analytical  models  of  stochastic  systems  are  one  example.  RSM  is  concerned  with  systems 
that  are  not  tmderstood  weU  enough  to  allow  for  this  approach,  hence  the  term  empirical  model 
building.  RSM  encompasses: 

1.  Designing  a  series  of  experiments  that  wiU  yield  adequate  and  reliable  measurements  of  the 
response(s)  of  interest  in  a  region  of  interest. 

2.  Analyzing  the  results  of  those  experiments  to  determine  a  mathematical  model  that  best  fits 
the  data  collected. 
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3.  Searching  for  the  optimal  settings  of  the  input  variables.  [27] 

Discrete  event  simulation  models  are  natural  candidates  for  an  RSM  study.  Simulation  models 
are  created  because  the  mathematical  relationships  that  govern  the  behavior  of  a  system  are  either 
not  imderstood  or  are  intractable.  Further,  the  output  of  a  simulation  model  is  a  random  variable. 
Since  RSM  studies  require  simulation  replications  at  numerous  experimental  design  points  and 
gradient  search  points,  the  studies  can  require  a  significant  amount  of  time  to  conduct.  Exactly 
for  those  same  reasons,  the  ACV  and  surrogate  search  could  reduce  that  time  appreciably.  Later 
in  this  dissertation,  these  time  saving  methods  are  demonstrated  on  an  example  RSM  study.  In 
order  to  apply  the  RSM  techniques,  a  brief  primer  on  RSM  will  be  presented  here.  The  primer  is 
primarily  adapted  from  Empirical  Model-Building  and  Response  Surfaces  by  Box  and  Draper  [12]. 
The  areas  discussed  include  empirical  models,  least  squares  analysis,  design  of  experiment,  steepest 
ascent,  second-order  model  fitting,  and  exploration  of  maxima  and  ridge  systems. 

2,4^ i  Empirical  Models.  In  the  introduction,  some  of  the  diflFerent  mathematical  mod¬ 
els  that  describe  systems  were  discussed.  RSM  outlines  a  means  of  deriving  a  statistical  model, 
or  empirical  model,  of  a  system.  The  assumption  is  that  there  exists  some  unknown  functional 
relationship 

=  (2-53) 

between  the  expected  response  of  the  system  under  investigation,  2/,  and  some  number,  k,  of  quan¬ 
titative  predictor  variables,  ^1,^2?  •  •  •  Since  this  relationship  is  unknown,  it  is  approximated 
by  a  poljmomial  approximation,  often  called  a  graduating  function^  over  a  specified  region  of  in¬ 
terest  [12].  In  deriving  the  graduating  function,  it  is  convenient  to  convert  the  input  variables  to 
coded  or  standardized  variables.  If  the  current  region  of  interest  is  defined  for  is  ^  where 
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IS  the  center  of  the  region,  the  coded  variable  Xi  is  defined  by 


OTi  =  (2.54) 

The  polynomial  graduating  function  of  the  coded  input  variables,  xi,X2j.  •  •  ,Xk  is  a  linear 
combination  of  powers  and  products  of  the  x’s.  A  polynomial  term  is  of  order  j  if  it  contains 
the  product  of  j  of  the  x’s,  where  some  or  all  may  be  repeated.  In  other  words,  the  terms 
xf,ariX2X3,x|x4  are  all  of  order  4.  A  polynomial  is  of  order,  or  degree,  d  if  the  highest  order 
term  in  the  polynomial  is  of  order  d.  The  general  form  of  the  polynomial  graduating  function  for 
fc  =  2  is  given  by 


g{  x.,/3)  —  /3o  +  {01^1  +  02^^)  d"  {011^1  +  022^1^2  +  ^12^2) 

+  {0111^1  "b  0222  ^2  +  0112^1^2  +  ^122^1^2)  +  ■  ■  •  (2.55) 

where  the  /3’s  are  coefficients  or  (empirical)  parameters  that  are  estimated  from  the  data. 

Box  and  Draper  [12]  point  out  that  the  polynomial  graduating  function  of  degree  d  can  be 
considered  a  Taylor’s  series  expansion  of  the  true  imderlying  function  /  (  ^)  trimcated  after  terms 
of  degree  d.  They  state  that  the  following  will  usually  be  true: 

1.  The  higher  the  degree  of  the  approximating  function,  the  closer  the  Taylor  series  can  approx¬ 
imate  the  true  function. 

2.  The  smaller  the  region  over  which  the  approximation  is  made,  the  better  the  approximation 
for  a  polynomial  function  of  a  given  order. 
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RSM  is  normally  concerned  with  only  first-order  and  second-order  graduating  functions.  For 
k  =  2  predictor  variables  the  first-order  graduating  function  is  given  by 

p(  X,  /3)  =  /3o  +  Z?!®!  +  /?2®2  (2-56) 

and  the  second-order  graduating  function  is  given  by 

g{  X,  0)=l3o+  PiXi  +02^2+  Piix\  +  ^22A  +  ^12^1^"^  (2-57) 

24.2  Least  Squares  Analysis.  Least  squares,  or  regression,  analysis  is  a  well-known 
statistical  method  of  fitting  empirical  functions  to  data.  Least  squares  analysis  is  used  to  fit  the 
first  and  second-order  graduating  functions  necessary  to  perform  a  RSM  study.  At  this  point  in  the 
discussion,  only  first-order  graduating  functions  are  considered  since  that  is  the  beginning  point 
of  almost  aU  RSM  studies  [12].  Since  the  theory  is  so  well  known  only  selected  highlights  of  least 
squares  analysis  and  how  they  apply  to  RSM  will  be  presented  here.  Box  and  Draper  [12]  provide 
an  excellent  section  on  least  squares  analysis.  For  further  study,  Neter  [37]  is  a  thorough  reference 
on  the  theory  and  apphcation  of  least  squares  analysis. 

Box  and  Draper  point  out  that  in  performing  RSM,  an  analyst  is  attempting  to  elucidate 
some  model 


y  =  f($,d)  +  e 


(2.58) 


where 


(2.59) 
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is  the  expected  level  of  the  response  y  given  the  k  predictor  variables  (li,  •  •  •  ?  ^ 

tion,  there  are  p  parameters  (^i ,  ^2  ?  •  *  •  stnd  e  is  the  experimental  error.  To  investigate  this 

model,  the  analyst  performs  a  series  of  experiments  at  n  different  settings  observing 

the  corresponding  2/i,2/2)  • « •  iVn  values  of  the  response.  There  are  two  important  questions  about 
the  resulting  function  that  must  be  answered: 

1.  Does  the  suggested  model  adequately  represent  the  data? 

2.  Assuming  the  model  does  adequately  model  the  data,  what  are  the  best  estimates  of  the 
model  parameters? 

In  practice,  the  second  question  is  addressed  first  using  the  method  of  least  squares  to  estimate 
the  model  parameters.  The  method  of  least  squares  selects  the  best  estimates  of  0  that  minimizes 
the  sum  of  squares  of  the  errors  given  by 


Sie)  =  ^[yu-fi^u,  of  (2-60) 

U=1 

5  (  0)  is  referred  to  as  the  sum  of  squares  function.  For  any  given  choice  of  p  parameters  of  0,  there 
is  a  specific  value  of  5  {  0).  The  minimizing  choice  of  0  is  the  least  squares  estimate  and  is  denoted 
by  0.  As  long  as  the  experimental  errors  eu^yu—f(^iu^  statistically  independent,  with 

constant  variance  and  normally  distributed,  the  least  squares  estimate  is  a  ‘‘good”  estimate  [12], 
Both  Box  and  Draper  [12]  and  Neter  [37]  provide  the  formulas  necessary  to  find  the  least  squares 
estimate  of  the  empirical  model. 

The  first  question,  “does  the  suggested  model  adequately  represent  the  data” ,  is  answered  in 
many  different  ways.  Essentially,  calculations  and  statistical  tests  can  be  performed  to  assess  model 
adequacy.  The  primary  guides  to  model  adequacy  are  the  size  of  the  mean  square  error  (MSE), 
the  coefficient  of  multiple  determination  (R^),  and  the  F  test  for  regression  relation. 
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MSE  is  defined  in  the  following  manner.  Prom  least  squares  analysis,  the  relationship 
between  the  vector  of  observed  responses,  (j/i,2/2) •  •  •  j2/n)  =  5  the  vector  of  fitted  responses 

*  •  -  j  Vn)  =  y  the  vector  of  residuals  (ei,  62, . . .  ,  e„)  =  e'  is  given  by 

e=  y-  y  (2.61) 

It  turns  out  that  the  vectors  y  and  e  are  orthogonal  to  each  other  [12],  so  that  e'  y  =  y'  e  =  0. 
This  results  in  the  relation 

y'  y  =  y'  y  +  e'  e  (2.62) 

so  that  the  sum  of  square  errors  (SSE)  is  given  by 

SSE  =  e'  e  =  y'  y  -  y'  y  (2-63) 

Then  MSE  is  given  by  SSE/(7i  —  p)  where  n  is  the  number  of  observations  and  p  is  the  number 
of  parameters  estimated.  MSE  is  useful  in  assessing  model  adequacy  since  it  is  a  measure  of  the 
variability  within  the  residuals  and,  thus,  provides  a  measure  of  how  well  the  fitted  responses  match 
those  actually  observed  [37].  Therefore,  the  smaller  the  value  of  MSE,  the  better  the  model. 

The  next  measure  for  the  adequacy  of  the  fitted  model  discussed  is  the  coefficient  of  multiple 
determination,  given  by 

B?  =  -  (2.64) 

'f^iyi-yf 

i—\ 

It  can  be  shown  that  0  <  <  1  and  that  represents  the  proportion  of  the  variability  within 

the  observed  responses  that  can  be  ejcplained  or  accounted  for  by  the  model  [12].  To  tmderstand 
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this,  first  consider  that  the  average  of  the  observed  responses  is  equal  to  the  average  of  the  fitted 
responses  [37].  Using  this  fact,  express  by 

/{n-i)  2 

R?  =  ^ ^  (2.65) 

Y^iVi-yf/n-l  ^ 

i=l 

SO  that  it  is  evident  that  is  the  ratio  of  the  sample  variance  of  the  fitted  values  (a  measure  of  the 
variability  within  the  responses  as  explained  by  the  model)  to  the  sample  variance  of  the  observed 
values  (a  measure  of  the  total  variability  within  the  observed  responses).  Hence  the  closer  the  value 
of  to  1,  the  more  the  analyst  can  assume  the  model  adequately  fits  the  data. 

Recalling  the  assumption  that  experimental  error  terms  are  independently  and  identically 
distributed  normal  random  variates,  a  statistical  test  for  model  adequacy  can  be  made.  It  turns 
out  that  if  this  assumption  holds,  the  ratio  of  the  mean  square  regression,  (  y  —  “  1)? 

the  mean  square  error,  e'  e/(n  —  p)  has  an  F  distribution  with  p  —  1  and  n-p  degrees  of  freedom, 
where  n  is  the  number  of  observations  and  p  is  the  number  of  parameters  estimated.  The  statistical 
test  is  posed  in  the  following  manner: 

Hq:  0  =  o(0i=02  =  -*  =  O) 

Hi  :  0  ^  0  (at  least  one  Oi^O]  i  =  1, 2, . . .  ,p)  (2.66) 

Then  the  test  statistic,  F  =  SSR/MSE  is  compared  to  the  critical  value  Fa,p-i,n-p-  If  the  test 
statistic  exceeds  the  critical  value,  the  null  hypothesis  is  rejected  and  the  conclusion  is  that  the 
fitted  model  is  significant. 

In  summary,  least  squares  analysis  provides  for  the  best  means  of  estimating  the  parameters 
of  the  graduating  function  and  the  size  of  MSE,  and  the  F  test  are  the  three  primary  measures 

of  model  adequacy.  Therefore,  a  “good”  model  wiU: 
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•  be  significant,  as  indicated  by  a  ‘large”  value  of  the  F  test  statistic; 

•  have  a  “small”  MSE;  and 

•  have  a  large 

Other  statistical  tests  exist  for  testing  the  significance  of  each  of  the  estimated  parameters 
and  lack  of  fit  for  further  model  testing  and  perfecting.  Each  is  described  in  Box  and  Draper  [12] 
and  Neter  [37]. 

2-4^3  Design  of  Experiment  The  graduating  function  relates  the  response  and  predictor 
variables  of  the  system  under  study.  Least  squares  analysis  details  the  method  of  constructing 
the  graduating  function  given  some  experimental  data.  The  logical  next  step  is  to  design  the 
experiments  that  are  to  be  conducted  in  order  to  collect  the  data.  Many  decisions  must  be  made 
when  designing  the  experiments.  These  include  the  number  of  experiments  and  the  levels  of  the 
predictor  variables  for  each  of  the  experiments.  Since  each  experiment  involves  the  use  of  valuable 
resources  (money,  equipment,  raw  materials,  labor,  and  time  are  some  examples)  one  of  the  primary 
goals  of  any  experimental  design  is  efficiency.  In  other  words,  the  analyst  wishes  to  perform  the  least 
number  of  experiments  that  wiQ  result  in  an  empirical  model  of  sufficient  accuracy.  In  addition,  the 
analyst  will  normally  wish  to  interact  fireely  with  the  data,  to  make  comparisons,  find  similarities, 
and  to  identify  trends.  One  class  of  experimental  designs  that  meets  these  requirements  is  the 
factorial  experimental  designs.  They  possess  the  following  properties  [12]: 

1.  They  allow  for  numerous  comparisons  and  so  facilitate  model  creation  and  criticism. 

2.  They  produce  estimates  of  the  parameters  whose  variance  is  as  small,  or  nearly  so,  as  those 
produced  by  any  design  covering  the  same  region,  hence  they  are  highly  efficient. 

3.  The  parameter  estimates  are  easy  to  calculate. 
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Factorial  designs  will  be  discussed  in  some  detail  below  followed  by  a  short  discussion  on  two 
other  types  of  experimental  design,  blocking  and  fractionating,  that  are  based  on  factorial  designs. 
Finally,  some  of  the  other  possible  exploratory  designs  will  be  mentioned. 

One  factorial  design  that  is  especially  useful  at  the  exploratory  stage  of  a  study,  when  not 
much  is  known  about  the  system,  is  the  twoAevel  factorial  design  [12].  They  are  also  useful  as  a 
first  building  block  for  developing  many  other  experimental  designs.  Two-level  factorial  designs 
are  designated  by  2^,  where  k  is  the  number  of  predictor  variables  and  2^  is  the  number  of  design 
points.  Often  the  predictor  variables  that  are  changed  during  the  course  of  experimentation  are 
referred  to  as  factors.  If  we  consider  a  design  point  as  the  particular  combination  of  levels  for  all 
the  factors,  the  factorial  design  consists  of  all  2*  design  points  where  each  predictor  variable  level 
is  set  to  one  of  two  levels.  Using  the  coded  predictor  variables,  this  can  be  represented  by 

=  (±1,±1, ...  ,±1)  (2.67) 

where  every  possible  combination  of  it  signs  is  selected  in  turn.  The  design  can  be  thought  of 
geometrically  where  each  design  point  is  one  of  the  vertices  in  a  dimensional  hypercube.  The 
designs  are  often  listed  in  what  is  called  standard  order  when  designing  the  experiments  or  listing 
the  results.  One  way  of  doing  this  is  listing  -1  and  4*1  alternatively  in  the  xi  column  for  a  total  of 
2^  times.  Under  the  X2  column  list  alternate  -1  -1  and  +1  +1  pairs,  under  the  x^  colunm  alternate 
fours  of  -1  -1  -1  -1  and  +1  4-1  4*1  +1  and  so  on.  A  standard  order  listing  for  a  2^  factorial  design 
is  shown  in  Table  2.1. 

The  main  effect  for  a  given  predictor  Variable  is  defined  as  the  average  difference  in  the 
response  level  as  the  level  of  the  predictor  variable  is  changed  from  low  to  high.  Factorial  designs  also 
allow  for  the  calculation  of  interaction  effects  as  weU  as  that  of  the  main  effects.  Interaction  between 
factors  occurs  when  the  difference  in  response  between  the  levels  of  one  factor  is  not  the  same  at  aU 
levels  of  the  other  fectors.  Box  and  Draper  [12]  demonstrate  a  simple  means  of  calculating  these 
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Table  2.1  Standard  order  table  for  2^  factorial  design. 


Xi 

X2 

-1 

-1 

-1 

+1 

-1 

-1 

-1 

+1 

-1 

+1 

+1 

-1 

-1 

-1 

-1-1 

-t-1 

-1 

■fi 

-1 

+1 

-1-1 

+1 

+1 

-t-i 

effects  using  a  table  of  signs.  The  table  is  easily  constructed  by  beginning  with  a  column  of  2*'  +l’s. 
The  next  k  columns  are  from  the  standard  order  table  labeling  them  1, 2, ...  ,  k.  Next  obtain 
the  (2*  -  A;  -  1)  interaction  columns  12, 13, ...  ,  123, . . .  ,  k  by  multiplying  the  signs  row  by  row 
as  indicated  by  column  headings.  At  the  bottom  are  written  the  divisors,  2*  for  the  first  column 
and  2*“^  for  all  the  others.  The  effects  (both  main  and  interaction)  are  then  calculated  by  adding 
the  responses  using  the  signs  from  the  appropriate  colmnn  and  dividing  by  the  appropriate  divisor. 
Such  a  sign  table  for  a  2®  design  is  illustrated  in  Table  2.2.  Although  there  are  quicker  means  of 
calculating  the  effects  [12]  and  in  most  cases  analysts  resort  to  computer  programs  for  calcxilation 
purposes,  the  table  of  signs  that  they  describe  is  useful  in  imderstanding  the  nature  of  the  various 
factorial  effects. 


Table  2.2  Colunms  of  signs  and  divisors  for  2®  factorial  design. 


Rim 

I 

1 

2 

3 

12 

13 

23 

123 

y 

1 

+1 

-1 

-1 

-1 

+1 

-hi 

-hi 

-1 

2/1 

2 

-t-i 

+1 

-1 

-1 

-1 

-1 

-hi 

-hi 

2/2 

3 

+1 

-1 

+1  ■ 

-1 

-1 

-hi 

-1 

-hi 

2/3 

4 

+1 

+1 

-fl 

-1 

-hi 

-1 

-1 

-1 

Vi 

5 

-1-1 

-1 

-1 

-1-1 

-hi 

-1 

-1 

-hi 

ys 

6 

-1-1 

-1-1 

-1 

+1 

-1 

-hi 

-1 

-1 

ye 

7 

-1-1 

-1 

-i-1 

4-1 

-1 

-1 

-hi 

-1 

y7 

8 

-1-1 

-1-1 

-|-1 

-hi 

-hi 

-hi 

-hi 

-hi 

ye 

Divisor 

8 

4 

4 

4 

4 

4 

4 

4 
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It  should  be  pointed  out  that  the  estimated  least  squares  coefficients  (parameters)  in  a  fitted 
first  order  polynomial  graduating  function  are  exactly  one  half  of  the  main  effect  as  defined  above. 
The  one  half  factor  relationship  occurs  because  the  main  effect  measures  the  response  change  to  a 
change  of  2  units  of  the  coded  variable  (-1  to  +1).  The  regression  coefficient  measures  the  change 
in  response  when  the  coded  variable  changes  by  1. 

If  it  is  possible  to  randomly  allocate  all  of  the  experimental  material,  resources,  and  the  order 
of  the  individual  runs,  the  experimental  design  is  considered  to  be  fully  randomized.  It  is  not  always 
possible  or  practicable  to  do  this.  For  example,  the  raw  material  may  be  suspected  of  inhomogeneity 
or  the  runs  may  be  accomplished  on  different  machines  or  by  different  techmcians.  These  differences 
could  cause  bias  between  the  different  set  of  circumstances,  or  blocks.  By  assuming  that  that  the 
differences  between  blocks  will  cause  the  response  to  simply  raise  or  lower  by  a  fixed,  unknown 
amount,  a  means  of  avoiding  the  bias  is  possible  [12].  This  is  commonly  referred  to  as  blocking. 
Blocking  introduces  a  new  factor  into  the  experiment,  the  block  factor,  which  indicates  the  block 
that  a  particular  rim  is  performed.  If  the  blocking  factor  is  not  accounted  for  in  the  experimental 
design,  its  effect  will  be  confounded  or  aliased  with  one  or  more  of  the  other  effects.  In  other  words, 
the  analyst  won’t  be  able  to  teU  whether  the  aliased  effects  are  caused  by  the  blocking  factor  or 
the  factors  aliased  with  the  blocking  factor. 

The  idea  in  blocking  is  to  divide  a  factorial  design  into  blocks  of  equal  size  and  then  aliasing 
the  blocking  factor  with  one  or  more  of  the  effects  that  are  not  “important”  or  expected  to  be 
significant.  The  factor  that  is  aliased  with  the  blocking  factor  is  caQed  the  blocking  generator. 
Simply,  the  blocks  are  constructed  by  gathering  the  runs  of  the  same  sign  of  the  blocking  generator 
into  the  separate  blocks.  This  can  be  illustrated  by  blocking  the  2^  factorial  design  in  Table  2.2 
into  two  equal  blocks  using  the  123  factor  as  the  blocking  generator.  This  is  shown  in  Table 
2.3.  Assuming  that  the  blocking  effect  is  additive,  it  can  be  shown  that  using  this  scheme,  that 
the  estimates  of  all  of  the  effects,  except  123,  is  unchanged  by  the  blocking  effect  [12].  However, 


2-32 


the  blocking  effect  and  the  123  interaction  effect  are  indistinguishable,  or  confounded.  As  long 
as  the  experimenter  is  concerned  with  only  the  main  effects  and  the  two-way  interaction  effects, 
the  illustrated  blocking  scheme  is  effective.  Box  and  Draper  [12],  and  many  other  sources,  provide 
tables  for  choosing  block  generators  for  2^  factorial  designs.  The  blocking  designs  are  constructed 
so  that  all  the  main  effects  are  not  aliased  with  blocking  effect  (s)  and  the  aliasing  that  does  occur 
is  with  the  least  number  of  high  order  interaction  terms. 


Table  2.3  2^  factorial  design  in  two  blocks. 


BlockI 


Run 

1 

2 

3 

12 

13 

23 

123 

1 

-1 

-1 

-1 

+1 

+1 

+1 

-1 

4 

+1 

+1 

-1 

+1 

-1 

-1 

-1 

6 

+1 

-1 

+1 

-1 

+1 

-1 

-1 

7 

-1 

+1 

+1 

-1 

-1 

+1 

-1 

Blockll 

Run 

1 

2 

3 

12 

13 

23 

123 

2 

+1 

-1 

-1 

-1 

-1 

+1 

+1 

3 

-1 

+1 

-1 

-1 

+1 

-1 

+1 

5 

-1 

-1 

+1 

+1 

-1 

-1 

+1 

8 

+1 

+1 

+1 

+1 

+1 

+1 

+1 

Often,  the  analyst  is  concerned  with  reducing  the  number  of  runs.  Each  run  costs  time  and 
money  and  the  number  of  runs  required  for  a  full  factorial  design  increases  exponentially  as  the 
number  of  factors  are  increased.  Fractional  factorial  designs  are  a  method  of  reducing  the  number 
of  experimental  runs.  Suppose  an  analyst  wanted  to  reduce  the  number  of  runs  for  a  2^  factorial 
design.  To  accomplish  this,  suppose  the  analyst  performed  only  those  runs  described  Block  11  in 
Table  2.3.  This  is  called  a  half  fraction  of  the  2^  factorial,  designated  as  a  2^“^  design.  Notice 
that  columns  1  and  23  axe  identical,  along  with  columns  2  and  13  and  columns  3  and  12. 
Therefore,  each  of  these  paired  effects  are  indistinguishable  from  each  other.  The  same  is  true  for 
Block  I,  the  difference  being  that  the  signs  are  reversed  for  each  confounded  column.  Thus,  for  a 
half  fraction  of  the  2^  factorial,  the  two-way  interactions  are  confounded  with  the  main  effects,  ff  it 
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is  known,  or  expected,  that  the  two-way  interactions  are  insignificant,  this  is  a  satisfactory  design 
for  estimating  the  main  effects.  If  not,  the  full  factorial  design  must  be  performed. 

The  resolution  of  a  factorial  design  describes  the  amount  of  confoimding  within  the  design. 
To  define  resolution,  begin  with  the  defining  relation  which  is  the  set  of  all  effects  (or  words)  that 
are  equivalent  to  the  identity  within  a  fractional  design.  Here,  identity  refers  to  the  generating 
effect  that  is  composed  of  dl  “l”’s  (all  positive  or  all  negative).  The  resolution  is  then  defined 
as  the  length  of  the  shortest  word  in  the  defining  relation.  Hence,  the  2^“^  half  fraction  design 
described  above  is  of  resolution  III,  since  the  defining  relation  consist  of  only  the  123  effect.  It 
can  be  shown  that  that  designs  with  the  following  resolution  have  the  following  properties  [12]: 

•  Resolution  HI  No  main  effects  are  aliased  with  any  other  main  effects,  but  the  main  effects 
are  aliased  with  two-factor  interactions  and  two-factor  interactions  may  be  aliased  with  each 
other. 

•  Resolution  IV  No  main  effects  are  aliased  with  any  other  main  effects  or  with  two-factor 
interactions.  The  two-factor  interactions  are  aliased  with  each  other. 

•  Resolution  V  No  main  effects  are  aliased  with  any  other  main  effects  or  with  any  two-  or  three- 
factor  interaction  effects.  Two-way  interaction  effects  are  aliased  with  three-way  interactions 
or  higher. 

Based  on  this,  Box  and  Draper  [12]  point  out  that  since  first-order  polynomials  are  used  to 
construct  the  exploratory  empirical  models,  designs  of  resolution  HI  or  higher  should  be  used.  That 
way,  no  main  effect  is  aliased  with  any  other  main  effect,  although  they  may  be  aliased  with  two- 
factor  interactions.  If  it  is  assumed  that  the  model  can  be  adequately  represented  by  a  first-order 
polynomial,  the  two-factor  interactions  are  assumed  to  be  zero  anyway. 

Many  other  experimental  designs  are  available  to  the  analyst  for  conducting  exploratory 
experiments.  Among  the  commonly  used  designs  are  foldover  designs,  saturated  designs,  and 
Plackett  and  Burman  designs.  Each  of  these  are  designs  that  are  other  than  full  fractions  that 
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meet  certain  aliasing  requirements.  Several  resources  are  available,  including  Box  and  Draper  [12], 
so  that  the  analyst  can  design  an  experiment  that  will  most  efficiently  fulfill  the  model  building 
requirements. 

244  Steepest  Ascent  Up  to  this  point,  the  discussion  has  focused  on  techniques  for 
constructing  first  order  polynomials  over  some  portion  of  the  operability  region,  as  opposed  to 
the  entire  region.  In  most  cases  the  entire  region  is  not  explored  at  once,  even  if  the  full  extent 
of  the  region  is  known.  First  a  complex  polynomial  may  be  necessary  to  describe  the  response 
surface  it  encompasses  and  an  excessively  large  number  of  runs  would  have  to  be  performed  to 
fit  such  a  polynomial.  Secondly,  often  large  regions  of  the  operability  region  may  be  known  to  be 
uninteresting  or  unprofitable.  Since  the  goal  of  most  RSM  studies  is  to  maximize  (or  minimize) 
some  response,  after  fitting  a  first  order  polynomial  in  the  initial  region  of  exploration,  the  next 
step  is  to  perform  a  search  outside  this  region  to  locate  a  point  where  the  response  is  maximized 
(or  minimized),  at  least  locally.  For  the  rest  of  this  discussion,  only  maximization  will  be  referred 
to  since  the  procedure  for  finding  a  minimum  is  the  same,  only  in  the  opposite  direction.  Once 
this  maximum  is  located,  a  new  experiment  can  be  designed  and  conducted  to  search  for  the  global 
maximum.  This  is  done  with  either  another  first  order  polynomial,  or  a  second  order  polynomial 
if  it  becomes  significant. 

In  Box  and  Draper’s  opinion  [12],  the  most  effective  and  efficient,  in  terms  of  number  of  runs, 
means  of  locating  the  local  maximum  is  to  use  the  method  of  steepest  ascent.  Given  that  a  first 
order  polynomial  has  been  fitted  to  the  data,  a  vector  at  the  center  of  the  region  that  makes  a  right 
angle  to  the  planar  contours  of  the  response  and  points  in  the  direction  of  increasing  response  is  the 
direction  of  steepest  ascent.  Prom  calculus,  this  vector  is  known  as  the  gradient.  The  gradient  points 
in  direction  of  steepest  ascent.  For  a  given  differentiable  multivariable  function  /,  the  gradient  at 
a  point  a  =  (ai,a2, . . .  ,afe)  is  defined  as  the  vector  whose  elements  are  the  partial  derivatives 
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evaluated  at  a  represented  by 


V/(a)=(^(«.),^fe).....^K))'  (2.68) 

For  a  first  order  model,  these  partial  derivative  values  are  simply  the  coefficients  of  the  main  effects. 
A  unit  gradient  vector  is  the  vector  of  length  one  in  the  direction  of  the  gradient  vector.  Although 
the  predicted  response  of  a  fitted  model  is  invariant  to  scale  changes  over  a  fixed  design  region, 
according  to  Box  and  Draper  [12],  the  direction  of  the  gradient  does  vary  according  to  the  scaling 
scheme  selected.  They  point  out  that  the  steepest  direction  of  ascent  is  calculated  when  using  the 
units  of  design  scaling  and  recommend  its  use. 

The  scheme  now  is  to  perform  experiments  from  the  center  of  the  initial  experimental  design 
in  the  direction  of  the  gradient  until  the  maximum  response  on  that  path  is  observed.  Some 
decisions  that  must  be  made  include  selection  of  appropriate  step  size  and/or  search  strategy  along 
the  steepest  ascent  path.  Another  decision  is  whether  to  move  in  the  direction  of  all  or  just  the 
significant  parameters.  Finally,  it  must  be  decided  whether  to  use  the  observed  maximiim  along 
the  path  or  estimate  it  based  on  a  fit  to  the  data  collected.  No  hard  and  fast  rules  exist  for  these 
decisions.  It  is  left  to  the  analyst  to  make  these  decisions  based  on  his/her  best  judgement  based 
on  observations  of  the  data,  past  experience,  and  any  previous  knowledge  of  the  system.  As  Box 
and  Draper  [12]  point  out,  it  should  be  remembered  that 

a  subject  as  concrete  and  mathematically  satisfying  as  experimental  design  is  actually 
embedded  in  a  morass  of  uncertainty,  uncertainty  due  to  the  possibilities  that  the  ex¬ 
perimenter  might  choose  wrong  variables,  might  explore  the  wrong  region,  or  might  use 
scaling  that  was  inappropriate. 

They  do  point  out  that  all  is  not  lost  however.  First,  analysts  often  know  (or  can  find  out)  a 
great  deal  about  the  system  imder  study.  Also,  since  an  RSM  study  is  conducted  sequentially,  the 
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analyst  doesn’t  have  to  be  exactly  right.  Being  “sufficiently”  close  to  right  will  get  one  on  one  of 
the  many  possible  paths  to  the  right  answer. 

Other  issues  discussed  by  Box  and  Draper  [12]  include  steepest  ascent  subject  to  a  constraint 
and  the  confidence  region  for  the  direction  of  steepest  ascent.  Simply  put,  when  performing  a 
steepest  ascent  search  and  a  constraint  is  encountered,  the  factor  that  is  constrained  is  held  at 
that  level  while  the  search  is  continued  along  the  gradient  directions  for  the  other  factors.  Box  and 
Draper  [12]  show  that  the  confidence  region  for  the  gradient  is  represented  geometrically  by  a  cone 
whose  vertex  is  at  the  center  of  the  design  region.  A  good  indication  that  the  direction  of  steepest 
ascent  has  been  determined  accurately  enough  is  the  magmtude  of  the  solid  angle  of  the  confidence 
cone  about  the  estimated  vector.  Box  and  Draper  [12]  provide  a  method  for  using  the  t-table  for 
assess  the  size  and  confidence  of  this  angle. 

2.4.5  Second- Order  Model  Fitting.  At  some  point  in  an  RSM  study  the  first-order  empir¬ 
ical  model  may  not  adequately  represent  the  system  under  study.  Reasons  include  insignificance  of 
the  fitted  first-order  model,  significant  lack  of  fit,  and/or  significant  higher  order  terms.  Given  that 
the  model  is  inadequate.  Box  and  Draper  [12]  describe  two  methods  of  finding  an  adequate  model 
for  the  system.  The  first  method  involves  transforming  the  response  or  predictor  variables  so  that 
the  system  can  be  adequately  described  by  a  first-order  model  of  the  transformed  variables.  The 
second  approach  is  to  perform  additional  experimental  runs  in  order  to  test  for  quadratic  terms  and 
then  fit  a  second-order  model  if  appropriate.  This  section  briefly  discusses  each  of  these  approaches, 
with  an  emphasis  on  the  second.  Finally,  the  section  includes  a  discussion  on  experimental  designs, 
based  on  two-level  designs,  that  enable  the  fitting  of  second-order  models. 

Box  and  Draper  [12]  point  out  that  the  metrics  (unit  of  measmrement)  used  to  record  data  are 
chosen  for  the  convenience  of  measurement.  However,  the  simpler  metrics  don’t  necessarily  result 
in  simpler  models.  Rather,  some  transformation  of  the  response,  predictor  variables,  or  both  may 
result  in  a  simpler  model.  Nonlinear  transformations,  such  as  the  square  root,  log,  or  reciprocal  of  a 
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response  effectively  expand  the  scale  in  one  part  of  the  range  while  contracting  it  in  another.  These 
transformations  are  called  the  power  transformations  [12].  Given  an  appropriate  transformation, 
a  first-order  model  may  still  be  used  to  adequately  model  the  system.  Box  and  Draper  describe 
analjdic  (predictive  score  function)  and  graphical  (residual  plots)  tests  for  determining  the  need  for 
transformation.  They  also  provide  procedures  for  finding  the  optimal  transformation  functions. 

Given  that  a  transformation  of  the  data  doesn’t  result  in  an  adequate  first-order  model,  a 
simple  statistical  test  for  the  pr^ence  of  quadratic  terms  is  described.  By  augmenting  a  two-level 
factorial  design  with  replications  at  the  center  of  the  design,  the  sum  of  squares  can  be  furthered 
partitioned  into  a  sum  of  squares  for  the  presence  of  pure  quadratic  terms  [12].  The  test  compares 
the  average  response  at  the  comer  points  of  a  two-level  factorial  design  with  that  at  the  center  of 
the  design.  To  describe  this  test,  let  Uf  represent  the  number  of  observations  made  at  the  comer 
points  of  the  original  2*^  or  2*’"*’  design  and  no  >  1  the  number  of  observations  (replications)  at  the 
center  of  the  original  design.  Then  the  average  response  at  the  corner  points  can  be  calculated  by 
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where  is  a  response  jfrom  one  of  the  comer  points.  The  average  response  at  the  center  point  is 
found  by 


Vno 


1  no 


(0) 


(2.70) 


with  a  response  from  the  center  point.  If  a  second-order  model,  given  by 


=  /^o  +  + - 1-  +  ^11*1  +  012X1X2  -I- - 1-  0kkxl  (2.71) 
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is  appropriate,  the  expected  value  of  the  average  response  at  the  comer  points  is  given  by 

[y«/]  —  "I - 1"  Pkk  (2.72) 

and  the  expected  value  of  the  average  response  at  the  center  point  is 

E\yno]=^o  (2-73) 

Therefore  E  -  y„o]  =  /?ii  -I - ^^kk  which  are  the  quadratic  terms.  So  there  is  an  indication 

that  the  “pure”  quadratic  terms  are  important  if  this  difference  is  significant. 

To  test  for  the  significance  the  ratio  SSPQ/MSPE  is  formed  where  SSPQ  is  the  sum  of  squares 
for  pure  quadratic  terms  and  MSPE  is  the  mean  square  error  for  pure  error.  SSPQ  is  computed  by 

SSPQ  =  (g«/  -  %o.)!  (2.74) 

^0  + 

and  has  one  degree  of  freedom.  To  find  MSPE,  the  square  error  for  pure  error,  SSPE  is  calculated 
by 


m  Ti 

SSPE  =  EE  {Viu-Vif  (2.75) 

i=l  u—1 

which  has  n  —  m  degrees  of  fireedom,  where  m  =  n/  + 1  is  the  number  of  design  points,  n  =  n/ + no 
is  the  number  of  observations  and  is  the  number  of  replications  at  the  design  point.  Thus 
MSPE  =  SSPE/(n  -  m). 
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Then  if  it  is  assumed  that  E  [vuf  “  Vno]  is  significant,  it  is  known  that  SSPQ/MSPE  has  a 
F  distribution  with  1  and  n  —  in  degrees  of  freedom.  The  statistical  test  is  then  formulated  as 

Ho  :  /3ii-/?22  =  -*  =  0 

Hi  :  at  least  one  /  0  z  ==  1, 2, . . .  fe  (2.76) 

so  that  if  SSPQ/MSPE  is  greater  than  a  value  of  an  F  distribution  with  degrees  of  freedom  1  and 
n  —  m,  the  nuU  hypothesis  is  rejected. 

Next,  the  focus  is  to  build  on,  or  augment  the  two-level  factorial  designs  for  first-order  models 
in  order  to  fit  a  full  second-order  model.  Although  the  center  point  replications  from  the  procedure 
described  above,  p>ermit  investigation  into  model  curvature,  the  additional  replications  don’t  permit 
for  estimation  of  a  full  quadratic  model  [12],  A  central  composite  design  (CCD)  builds  on  the  two- 
level  factorial  design  that  has  been  augmented  with  center  point  replications.  A  CCD  is  constructed 
by  adding  =  2k  axial  design  points  of  the  form 

(dia,  0, . . . ,  0),  (0,  ±a, , . . ,  0)  (0, 0, . . .  ,  ±a)  (2.77) 

where  a  is  usually  equal  to  with  n/  the  number  of  design  points  in  the  factorial  portion  of 
the  design.  The  CCD  has  several  properties  to  recommend  it,  in  addition  to  having  an  embedded 
two-level  factorial  design.  The  design  is  a  rotatable  second-order  minimum  bias  design.  Rotatable 
means  that  the  accuracy  of  the  predicted  response  is  a  function  of  only  the  distance  from  the  center 
of  the  design,  not  the  direction  [12]. 

Unless  the  analyst  expects  that  a  second-order  model  will  be  required,  runs  at  the  axial  design 
points  may  be  performed  after  the  results  from  a  two-level  factorial  design  are  analyzed.  In  that 
case,  a  blocking  effect  is  likely  the  result  of  the  sequential  experimentation.  In  that  case,  it  is  possible 
to  design  a  CCD  where  the  block  effects  can  be  estimated  separately  and  independently  from  those 
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of  the  other  factors.  This  type  of  design  is  said  to  blocked  orthogonally  and  is  accomplished  by  the 
selection  of  a  and  the  number  of  center  point  replications  performed  in  each  block  [12].  Since  this 
dissertation  is  concerned  with  performing  RSM  on  a  simulation  model,  it  is  not  necessary  to  be 
concerned  with  this  type  of  blocking  as  long  as  all  replications  are  independent. 

2.4-6  Exploration  of  Maxima  and  Ridge  Systems.  Given  a  fitted  second-order  model, 
the  next  task  is  to  search  for  a  maximum.  Box  and  Draper  [12]  describe  a  general  strategy  using 
canonical  analysis  that  is  summarized  here.  Canonical  analysis  is  an  approach  for  analyzing  the 
fitted  second-order  model  by  rotating  the  axes  to  remove  all  cross-product  terms  and  when  necessary 
translation  *of  the  coordinate  axes  to  coincide  with  the  stationary  point.  A  model  that  has  been 
rotated  and  translated  in  this  manner  is  said  to  be  in  canonical  form.  The  advantages  of  this  form 
allow  the  analyst  to  identify  a  local  optimum,  if  one  exists,  and  to  describe  the  response  surface  in 
a  straightforward  manner.  When  the  model  is  only  rotated,  that  is  referred  to  A  canonical  form. 
A  rotated  and  translated  model  is  said  to  be  in  B  canonical  form. 

The  following  matrix  notation  is  developed  for  use  throughout  this  section.  A  quadratic 
function  ^ven  by 


y  =  6o  +  biXi  +  •  *  ■  +  bjcXk  -f*  bi2XiX2  +  •  •  *  + 


(2.78) 


can  be  represented  in  matrix  form  as 


y  =  bo  +  x'b-f  x'  Bx 


(2.79) 


where  x  =  (a:i,a:2, . . .  ,Xfc)',  b  =  (6i, 62,  •  •  •  and 


bn 

2^12 

5612 

622 

^b2k 

2^1* 

2^2fe 

bkk 

(2.80) 


Prom  calculus,  it  is  known  that  a  necessary  condition  for  a  point  to  be  a  maximum  (minimum),  it 
must  be  a  stationary  point.  For  a  point  to  be  a  stationary  point,  the  first  derivative  (if  it  exists) 
evaluated  at  that  point  equals  zero.  For  the  second-order  model  described  here,  a  stationary  point 
can  be  found  (if  it  exists)  by  setting  the  derivative  of  y  equal  to  the  fc  x  1  zero  vector  and  solving. 
Since  the  derivative  of  y  is  given  by 


b  +  2x'  B 


the  stationary  point,  if  it  exists,  is  found  by  solving 


Xs 


(2.81) 


(2.82) 


K  a  stationary  point  exists,  the  next  step  of  the  general  strategy  is  to  determine  its  Euclidean 
distance  from  the  center  point  of  the  current  experimental  design.  This  is  computed  as 


(2.83) 


Based  on  !>,  determine  if  is  within  the  current  experimental  design  space.  If  it  is,  Box  and 
Draper  [12]  recommend  that  the  empirical  model  be  transformed  to  B  canonical  form  in  order  to 
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characterize  the  response  surface.  B  canonical  form  is  given  by 


y  =  ys+x'AX  (2.84) 

where  A  is  the  k  x  k  diagonal  matrix  whose  non-zero  elements  are  the  eigenvalues  of  B  and 
ys  is  the  value  of  the  response  at  the  stationary  point  (origin  of  the  new  coordinate  system). 
X  =  M'(  X  -  Xs)  where  Xs  is  the  stationary  point  and  M  is  fc  x  A:  orthonormal  matrix  whose 
columns  are  the  standardized  eigenvectors  of  B. 

If  D  indicates  that  the  stationary  point  is  outside  of  the  experimental  design  space,  or  if  no 
stationary  point  exists,  Box  and  Draper  recommend  putting  the  model  in  A  canonical  form  in  order 
to  characterize  the  response  surface.  This  is  accomplished  via 

2/ =  60+  e'  X-h  X' AX  (2.85) 

where  X  =  M'  x  and  6  =  M'  b.  M  and  A  are  the  same  as  defined  above.  With  the  model 
in  A  canonical  form,  the  next  step  is  to  perform  a  search  for  the  optimum  by  performing  either 
elucidation  of  a  ridge  system  or  ridge  analysis. 

A  stationary  point  in  a  second-order  model  far  from  the  center  of  an  experimental  design 
region  implies  the  existence  of  a  ridge  system  [12].  Elucidation  of  a  ridge  system  is  an  analytical 
means  of  detecting,  describing  and  exploiting  the  ridge  system.  To  perform  elucidation  of  a  ridge 
system,  after  transforming  the  model  to  A  canonical  form,  the  analyst  studies  four  measures  for 
each  factor.  These  measures  are 

•  6i  slope  for  factor  i 

•  Xi  quadratic  coefficient  for  factor  i 

•  Xis  distance  from  the  design  center  to  the  stationary  point  for  factor  i 
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•  Vi  approximate  range  of  y  due  to  factor  i  over  the  design  region  with  ri  =  y  30^  +  A? 

It  should  be  pointed  out  that  each  of  these  measures  are  statistical  estimates  of  the  true 
values.  Since  the  CCD  design  is  orthogonal  and  rotatable,  the  standard  error  of  the  original,  coded 
regression  coefficients  can  be  used  for  an  approximation  of  the  standard  errors  of  the  A  canonical 
coefficients  [12].  Based  on  these  measures,  the  ridge  is  estimated  and  a  direction  along  the  ridge  is 
determined  for  further  experiments.  The  details  of  the  method  are  outlined  in  Box  and  Draper  [12], 

Ridge  analysis  locates  the  maximum  estimated  response  that  is  a  distance  R  from  the  design 
center.  The  method  can  be  posed  as  the  following  constrained  optimization  problem 

Maximize  y  =  bo  +  x'b-h  x'  Bx 

subject  to  Yli=i  ^  R^  =0  (2.86) 

Although  Box  and  Draper  [12]  don’t  recommend  its  use,  several  computer  packages,  including 
SAS,  make  the  method  available.  In  the  computer  packages,  ridge  analysis  for  many  values  of 
R  can  be  rapidly  performed.  The  result  is  that  a  new  design  center  can  be  located  for  further 
experimentation. 
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III.  Simulation  and  Analytic  Modeling 


3.1  Overview 

Throughout  this  dissertation,  we  use  both  simulation  and  analytical  models  in  concert  to 
analyze  and  study  several  different  systems.  Because  the  methods  described  often  commingle  the 
inputs  and  outputs  of  both  t3^es  of  models  we  have  included  a  short  discussion  on  the  two  t3rpes  of 
models.  For  clarity,  the  notation  developed  here  will  be  used  throughout  the  rest  of  the  dissertation. 
We  begin  by  defining  the  terms  model  and  system  for  the  purposes  of  this  dissertation.  This  is 
followed  by  a  discussion  of  discrete  event  simulation  models  with  a  short  tutorial  on  one  method 
of  estimating  output  performance  measures  using  simulation  models.  We  then  present  a  discussion 
on  analytical  models. 

3.2  Systems  and  Models 

Essentially,  a  system  is  the  “thing”  imder  study.  According  to  Law  and  Kelton  [32],  “A 
system  is  defined  to  be  a  collection  of  entities,  e.g.,  people,  or  machines,  that  act  and  interact 
together  toward  the  accomplishment  of  some  logical  end.  ...  In  practice,  what  is  meant  by  the 
system’  depends  on  the  objectives  of  a  particular  study.”  Pritsker  [40]  describes  a  system  as  “a 
collection  of  items  from  a  circumscribed  sector  of  reality  that  is  the  object  of  study  or  interest.” 
Balci  states  simply,  the  term  “system  is  used  to  refer  to  the  entity  that  contains  the  problem  to  be 
solved”  which  can  have  inputs,  parameters,  and  outputs  [5]. 

We  adopt  the  simple  definition  offered  by  Balci  [5],  which  is  depicted  graphically  in  Figure 
3.1.  The  roughness  of  the  “system”  block  impHes  that  the  boundary  between  the  system  and  the 
rest  of  reality  is  not  easily  defined  and  depends  on  the  problem  statement  and  the  judgement  of  the 
analyst.  The  inputs  to  the  system  include  a  vector  of  structural  parameters  ^2?  *  *  * 

that  consists  of  those  things  input  to  the  system  that  don’t  change  over  time  or  space.  For  example, 
elements  within  could  include  the  number  and  tj^es  of  aircraft  in  a  particular  fleet  of  aircraft 
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or  the  number  of  refueling  hydrants  at  a  particular  air  base.  The  variable  input  vector,  0  = 
(^1,^2,  •••  contains  those  inputs  to  the  system  that  do  change  over  time  or  space.  Some 

examples  of  input  variables  include  the  time  required  to  service  an  aircraft  or  the  amoimt  of  fuel 
stored  at  a  particular  air  base.  The  output  vector  Y  =  (Yi,y2j  •  •  •  includes  values  that  may 
or  may  not  change  over  time  or  space  and  are  the  result  of  the  system  acting  upon  the  different 
inputs.  Each  element  of  the  output  vector  could  be  a  vector  of  several  occurrences  of  the  item 
tmder  interest  or  could  be  single  value.  Examples  of  outputs  include  aircraft  throughput  or  the 
times  required  to  service  aircraft  over  a  specified  amount  of  time. 


As  with  the  term  system,  there  are  several  different  definitions  for  a  model.  First  we  note  that 
we  are  referring  to  mathematical  models,  not  physical  models.  A  mathematical  model  represents  “a 
system  in  terms  of  logical  and  quantitative  relationships  that  are  then  manipulated  and  changed  to 
see  how  the  model  reacts,  and  thus  how  the  sj^tem  would  react — if  the  model  is  a  valid  one  [32].” 
More  simply,  a  model  “is  a  representation  of  a  system  with  inputs,  parameters,  and  outputs  [5].” 
There  are  several  different  types  of  mathematical  models.  In  the  next  sections  we  focus  on  two  of 
them — simulation  and  analytical  models. 
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S.3  Simulation  Models 


For  this  dissertation,  simulation  model  refers  to  a  discrete  event  srmtilation  model.  Such 
models  are  dynamic  in  that  they  represent  a  system  as  it  evolves  over  time,  discrete  in  that  the 
state  variables  that  describe  the  system  change  instantaneously  at  separated  points  in  time,  and 
stochastic  in  that  they  have  random  inputs  and  outputs  [32].  Due  to  the  size  and  complexity  of  most 
simulation  models,  they  are  normally  translated  to  computer  code  so  that  they  can  be  evaluated 
on  a  computer.  One  approach  to  defining  a  simulation  model  is  depicted  in  Figure  3.2.  In  this 
case  we  choose  to  define  a  simulation  model  in  an  analogous  manner  to  the  definition  of  a  system 
provided  in  Figure  3.1. 


Figure  3.2  Simulation  model  characteristics 


We  define  a  simulation  model  as  the  multivariable  function,  ^  9  ^ 

the  superscript  S  is  used  to  identify  the  function,  inputs,  and  outputs  as  part  of  the  simulation 
model.  As  before,  is  a  vector  of  structiural  parameter  inputs  that  in  this  case  do  not  change 
during  a  realization  of  the  simulation  model.  For  example,  these  structural  parameters  might 
represent  the  number  of  aircraft  or  parking  spots  in  the  system  being  modeled.  The  random 
variable  parameter  vector  6^  consists  of  the  parameters  that  specify  the  random  variables  that 
drive  the  simulation  model.  These  random  variates  could  represent  service  times  or  failure  rates 
for  example.  For  clarity,  we  have  located  the  random  variate  generator  (actually  psuedo-random 
variate  generator)  outside  of  the  simulation  model.  The  random  variate  generator  is  a  separate 
function,  p,  of  the  random  variate  parameter  vector,  0^  that  produces  an  output  vector,  Xi, 


for  replication  i  whose  elements  are  independently  and  identically  distributed  (IID)  stochastic 
processes.  Hence,  the  random  variate  generator  is  given  by  g  =  X^.  The  simulation  model 

then  transfoims  these  inputs  into  a  vector  of  output  stochastic  process  for  replication  %  represented 
by  Yf .  These  stochastic  processes  could  include  the  set  of  all  observed  repair  times  or  the  times  that 
aircraft  depart  an  airfield.  Note  that  most  of  the  output  stochastic  processes  are  not  independently 
nor  identically  distributed  [32].  Included  in  our  definition  of  Yf  are  the  realized  random  variate 
stochastic  processes  that  are  also  inputs  to  the  model  (which  are  IID). 

Since  the  outputs  of  a  simulation  model  are  stochastic  processes  that  are  not  HD,  several 
methods  of  simulation  output  analysis  have  been  developed.  We  present  the  method  of  independent 
replications  used  throughout  this  dissertation.  The  following  presentation  is  excerpted  from  Law 
and  Kelton  [32]  and  is  graphically  depicted  in  Figure  3.3.  To  implement  this  method,  n  independent 
replications  of  the  simulation  model  are  generated.  By  independent  replication,  we  mean  that 
is  statistically  independent  of  for  z  =  1, 2, . . .  ,  n;  fe  =  1, 2, . . .  ,  n;  and  i  ^k.  Assume  that  we 
are  attempting  to  estimate  the  mean  of  a  single  element,  or  performance  measure  of  Yf  given  by 
E[Y^]  =  /A.  Then  let  Yf  ^P(i2)  ’  *  •  *  » realized  output  stochastic  process 

for  the  performance  measure  of  interest  for  rephcation  i.  We  begin  by  finding  the  sample  mean  of 
each  replication  by 

m 

i  =  (3.1) 

This  is  shown  in  Figure  3.3  in  the  “Replication”  block  of  the  “Output  Analysis”  section.  Then  we 
take  the  overall  mean  in  order  to  estimate  /x  by 


fl  =  Y^=n-^Y,Yf  (3.2) 

i=l 
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This  step  is  depicted  in  the  “Overall”  output  analysis  block  in  Figure  3.3.  Since  each  replication  is 
independent  of  the  others,  each  Yp^i)  are  also  independent  of  each  other.  Therefore,  usual  statistical 
methods  can  be  applied  to  //  to  determine  confidence  intervals  or  other  statistical  tests.  We  also 
note  that  in  some  cases,  a  number  of  realizations  at  the  beginning  of  the  output  stochastic  process 
may  be  deleted  if  the  analyst  is  attempting  to  estimate  a  steady  state  value  [32]. 


Figure  3.3  Simidation  model  method  of  independent  replications. 


We  also  graphically  display  the  internal  control  variate  method  of  variance  reduction  for 
independent  replications  in  Figure  3.4.  The  control  variate  method  is  presented  in  some  detail  in 
Section  2.2.1  so  we  keep  our  comments  brief  here.  Mainly  we  wish  to  point  out  that  the  vector  of 
internal  control  variates,  are  actually  elements  of  Yf .  As  described  earlier,  the  realized  input 
stochastic  processes,  such  as  service  times  at  a  particular  service  center,  can  be  collected  as  an 
output  process  for  use  as  an  internal  control  variate.  Also  note  that  the  control  variate  method  is 
a  “post-processing”  method  in  that  it  is  accomplished  after  all  replications  are  generated. 


Figure  3.4  Simulation  model  method  of  internal  control  variates. 
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3.4  Analytical  Models 


Unlike  simulation  models,  analytical  models  provide  exact,  analytical  solutions  [32],  A  simple 
example  of  an  analytical  model  is  given  by  the  formula  V  =  IR  where  V  is  the  voltage,  I  the  current, 
and  R  the  resistance  in  a  series  electrical  circuit.  Other  analytical  models  can  be  extremely  complex 
and  require  a  computer  to  reach  a  solution  in  a  reasonable  amount  of  time.  All  analytical  models 
used  in  this  dissertation  are  models  of  closed  queueing  networks.  The  models  vary  in  complexity, 
but  aU  require  the  use  of  a  computer  to  calculate  solutions. 

We  graphically  depict  an  analytical  model  of  the  type  used  in  this  dissertation  in  Figure  3.5. 
As  with  the  simulation  model,  an  analytical  model  can  be  defined  as  the  function  ,  0^^  = 

where  the  inputs  to  the  model  are  separated  into  two  separate  vectors.  The  vector  of  structural 
parameters,  defines  the  structural,  or  topological  elements  that  define  the  network  modeled. 
Examples  include  the  number  of  customers,  customer  classes,  and  the  number  of  service  center, 
or  queues,  in  the  network.  The  random  variate  parameter  vector,  6^  define  the  parameters,  or 
moments,  of  the  random  variates  modeled.  These  might  be  the  mean  service  times  at  each  of 
the  service  centers  or  the  probabilities  associated  with  the  routing  of  customers  throughout  the 
network.  The  output  vector  of  this  type  of  model  consists  of  probabilities  and/or  mean  values 
for  any  number  of  performance  measures.  Station  throughput  or  the  probability  of  4  customers  at 
a  certain  service  center  are  examples  of  analytical  model  output. 


Figure  3.5  Analytical  model  characteristics. 
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IV.  Analytic  Control  Variate  Monte  Carlo  Method 

4.1  Overview 

The  control  variate  method  is  a  well-known  variance  reduction  technique  for  discrete  event 
simulation.  This  chapter  explores  the  variance  reduction  achieved  by  employing  an  external  analyt¬ 
ical  model  to  consolidate  multiple  input  random  variables  into  a  single  “analytical”  control  variate 
(ACV).  As  discussed  in  Section  2.2.5,  previous  researchers  have  found  that  this  approach  can  pro¬ 
duce  significant  variance  reduction,  but  the  resulting  point  estimate  of  the  simulation  response  may 
exhibit  unacceptable  bias.  In  this  chapter  a  general  Monte  Carlo  method  of  distribution  sampling 
for  resolving  the  bias  problem  is  developed  and  demonstrated  through  a  queueing  network  example. 
The  mean  and  variance  of  the  input  random  variable  used  to  produce  the  ACV  must  be  known  in 
order  to  apply  the  Monte  Carlo  method.  Additionally,  the  distribution  of  these  random  variables 
must  be  known  either  exactly  or  approximately.  In  many  cases,  this  requirement  is  not  difficult  to 
meet.  With  this  modification,  the  ACV  method  performs  favorably  when  compared  with  the  classi¬ 
cal  internal  and  external  control  variate  approaches.  To  demonstrate,  the  different  control  variates 
are  compared  using  confidence  interval  width,  realized  coverage,  and  estimated  mean  square  error 
(MSE)  using  a  queueing  network  example.  The  efficiency  of  the  ACV  method  is  also  compared  to 
that  of  the  uncontrolled  response. 

This  chapter  is  organized  in  the  following  manner.  It  begins  with  a  discussion  of  the  ACV 
method  and  a  description  of  the  Monte  Carlo  method  for  reducing  bias.  Included  is  a  discussion  on 
the  efficiency  of  the  method.  This  is  followed  by  a  description  of  the  queueing  network  and  different 
control  variates  used  to  compare  the  different  methods.  Finally  a  comparison  of  the  results  for  each 
of  the  control  variates  is  presented. 
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4-2  Analytical  Control  Variates 


4,2.1  ACV  Construction.  Previous  researchers  have  introduced  and  explored  the  concept 
of  an  ACV  [18,34,48,49,53,54].  This  variance  reduction  approach  can  be  considered  a  hybrid  of 
both  internal  and  external  methods  since  an  external  analytical  model  is  employed  to  produce  a 
new  random  variable  that  is  essentially  a  function  of  internal  (input)  random  variables.  An  ACV , 
Z,  is  generated  for  every  replication  of  a  simulation  model  using  the  following  general  scheme.  For 
each  replication,  compute  the  sample  means  of  the  realized  input  random  variables  of  the  simulation 
model.  Then  compute  the  value  of  the  analytical  model  using  those  sample  means.  That  analytical 
model  value  is  the  ACV  for  that  particular  replication. 

The  simulation  and  analytical  models  are  described  as  follows.  Recall  that  we  define  a  simu¬ 
lation  model,  for  replication  j,  as  the  function 

j  =  l,2,...,n  (4.1) 

where  is  the  vector  of  structural  parameters,  0^  is  the  vector  of  random  variate  parameters, 
g{-)  is  the  random  variate  generator,  and  Xj  is  the  vector  of  IID  stochastic  processes  generated 
during  the  replication  by  the  random  variate  generator.  We  consider  a  simulation  model  where 
we  wish  to  estimate  an  unknown  performance  measure,  E[Yp]  =  fip  that  is  a  function  of  m 
input  stochastic  processes  given  by  Xj  =  (Xij,X2j,...  ,Xmj)^  for  replication  j.  Then  the 
stochastic  process  generated  during  replication  j  is  given  by  X^j  =  where 

is  a  constant  dimension  determined  by  the  simulation  replication  stopping  rule.  Since  each 
Xi,-  is  an  lED  stochastic  process,  we  let  each  Xijk  be  a  realization  of  the  random  variable  Xi  for 
i  =  1, 2, . . .  ,  m,  j  =  1, 2, . . .  ,  n,  and  A:  =  1, 2, . . .  ,  n.  Assume  that  the  probability  structiue  of  each 
Xi,  including  the  values  E[Xj\  =  /^i  and  Var{Xi)  =  t  =  1, 2, . . .  , m  is  known. 

The  response  yielded  by  an  appropriate  analytical  model  of  the  same  stochastic  system  is 
essentially  a  function  of  the  moments  of  a  subset  of  the  same  input  random  variables  (ideally,  all 
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of  the  “influential”  inputs).  The  anal5i;ical  model  is  represented  as  the  function  =  Z, 

where  Z  is  the  same  system  performance  measure  estimated  by  E[Y^]^  and  <l>^  and  6^  are  the 
input  structural  and  random  variate  parameter  vectors.  1£  the  analytical  model  is  a  reasonably 
accurate  representation  of  the  system  under  study,  then  the  anal3dical  response  should  be  highly 
correlated  with  the  simulation  response. 

For  example,  consider  simulation  and  analytical  models  of  a  fast  food  drive-through  system 
with  two  windows  in  series.  Using  a  discrete  event  simulation  model,  an  appropriate  value  for 
E[Y^]  might  be  the  steady-state  mean  sojourn  time  experienced  by  a  random  customer.  The 
random  variables  Xi  and  X2  might  represent  the  customer  service  times  experienced  at  the  two 
windows.  Additionally,  Xs  might  represent  the  customer  interarrival  time.  The  corresponding 
analytical  model  might  employ  a  queueing  network  algorithm  to  calculate  Z,  the  analjrtical  mean 
sojourn  time.  Appropriate  moments  of  Xi,X2  and  Xs  would  serve  as  inputs  to  the  analytical 
model. 

An  ACV,  Zj,  is  constructed  for  each  replication  j  using  the  sample  first  moment  esti¬ 
mators  of  some  subset  of  the  realized  observations  of  ,Xmi-  Rather  than  using 

the  input  stochastic  processes  themselves,  we  consider  the  simulation  output  processes  = 

such  that  C  Xij  for  i  =  The  subscript  A  de- 

notes  those  outputs  of  the  simulation  model  that  are  used  as  inputs  to  the  analytical  model  for 
the  purposes  of  generating  an  ACV.  We  make  this  distinction  between  simulation  inputs  and 
outputs  for  two  reasons.  First,  when  estimating  steady  state  simulation  models  using  the  repli¬ 
cation/deletion  method,  a  certain  number  of  realizations  are  deleted  before  statistics  are  com¬ 
puted  so  that  we  are  not  operating  on  the  full  input  processes.  Secondly,  in  the  next  chap¬ 
ter  we  consider  inputs  to  the  analytical  model  for  generating  an  ACV  that  are  only  simulation 
outputs  and  don’t  have  a  corresponding  stochastic  process  input.  To  generate  ACV’s  perform 
n  indei)endent  replications  of  the  simulation.  For  each  replication  y,  the  simulation  produces 
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a  vector  of  output  realizations  for  each  of  the  m  input  random  variables.  Represent  these  re- 
alized  vectors  by  •  •  yYldjn))''^  *  =  -  3  =  1,2,...  ,n  and 

Vi  is  a  constant  dimension  determined  by  the  simulation  stopping  rtile.  Form  a  new  vector, 
Y^O)  =  (YA(ii)>  for  each  replication  j,  where 

YA%)  =  rr'l'Yi(y)  i  =  l,2,...  ,m;  j  =  l,2,...  ,n  (4.2) 

Then  each  ^  ^  ^  input  to  the  analytical  model,  so  that  is  the 

ACV  for  replication  j.  If  E  known,  we  construct  the  control  variate 

estimator  as  before.  The  analytically  controlled  estimate  of  fi  is  defined  as 

y^0)=yp-H^-f^z)  (4.3) 


where  Z  is  the  sample  mean  of  the  ACVs  for  all  n  replications  and  ^  is  estimated  as  in  Equation 
(2.15).  As  long  as  Z  and  Yp  are  strongly  correlated,  significant  variance  reduction  will  occur. 

4-2,2  Monte  Carlo  Method.  Finding  the  value  of  to  construct  an  unbiased  estimate  of  fi 
is  often  a  difficult  problem.  If  the  analytical  model  represents  a  linear  function,  then  the  linearity  of 
the  expectation  operator  guarantees  that  =  E  Y^(j))]  =  •  j/^m)^)* 

Since  the  relationship  is  normally  non-linear,  computation  of  in  this  manner  will  produce  a 
biased  result  for  both  /x^  and  the  analytically  controlled  estimate  of  /x.  An  alternative  numerical 
approximation  (say  fi^)  must  be  found  in  this  case.  It  is  shown  that  given  a  known  probability 
structure  for  the  input  random  variables,  can  be  accurately  determined  through  a  Monte  Carlo 
approach. 

The  value  of  is  estimated  using  a  general  Monte  Carlo  method  that  generates  a  series 
of  random  vectors  that  approximate  (in  distribution)  the  inputs  that  the  analytical  model  sees 
when  generating  the  ACV.  By  generating  enough  of  these  vectors  and  calcialating  the  value  of 
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the  analytical  model  evaluated  at  each  of  these  vectors,  an  accurate  approximation  of  can 

be  obtained.  Recall  that  for  every  replication  j,  the  elements  of  each  input  vector  = 

0^A(ijiv^Aiij2y  ■  have  known  expectation  and  known  variance  cr?.  Prom  this,  the 

input  to  the  analytical  model  for  replication  j,  Y^y))  is  formed  as  the  vector  of  sample  means  of 

Y^(i^.).  Thus,  E  [Yi(^.J  =  -  .Mm)',  for  j  =  1,2 . n.  If  each  Xi  is  reasonably  large, 

the  central  limit  theorem  can  be  applied  to  each  element  of  Therefore,  the  distribution  of 

each  can  be  approximated  with  a  normal  distribution  having  mean  and  variance  frf/ri 

(where  is  a  constant  dimension  of  determined  by  the  simulation  stopping  rule). 

To  calculate  first  generate  G  Monte  Carlo  vectors 

_  (\rMC  \rMC  -yMC  ^  .  o  —  10  G 

^A{j)  -  ^A(2i).  •  •  •  .  ^A{mj) )  .  J  -  i,  •  •  •  ,  tr 

where  each  is  an  independent  pseudo-random  variate  from  a  normal  distribution  having  mean 

/Xj  and  variance  Then  approximate  with 

=  (4.5) 

^  J  =  1 

The  ACV  Monte  Carlo  method  is  depicted  in  Figure  4.1  in  the  same  manner  as  that  presented  in 
chapter  III.  Given  that  /^(*)  is  non-linear,  fiz  =  f^z  as  G  — >  oo  if  and  only  if  each  Yj^^  is  sampled 
from  the  exact  distribution  of  .  If  the  random  variables  Xi ,  that  define  the  stochastic  processes 

are  not  independent,  any  of  several  weU-known  techniques  for  generating  dependent  random 
variates  may  be  applied  [32]  .  One  particular  approach  for  generating  dependent  variables  is 
demonstrated  later  in  this  chapter. 

4,2,3  Monte  Carlo  Method  Efficiency,  Determining  the  optimal  value  of  G  involves  a 
trade-off  between  accuracy  and  computational  efficiency.  To  achieve  a  desired  level  of  accuracy, 
one  can  apply  standard  statistical  techniques  to  determine  the  number  of  replications  necessary  to 
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Figure  4.1  ACV  Monte  Carlo  method  of  variance  reduction. 


construct  a  confidence  interval  of  any  width  about  an  estimate  of  /x^.  The  smaller  this  confidence 
interval,  the  less  likely  the  resulting  value  of  will  cause  exc^sive  bias  in  the  ACV-controUed 
response  estimate.  Based  on  the  results  fi:om  this  chapter,  no  detectable  bias  will  occur  if  the  confi¬ 
dence  interval  width  (CIW)  about  is  less  than  10  percent  of  the  CIW  about  the  ACV-controlled 
response  (for  typical  values  of  a).  However,  as  more  computer  time  is  expended  calculating  /x^,  less 
time  is  available  for  simulation  replication  and  ACV  production.  Consider  the  following  questions. 
Is  a  better  estimate  of  fi  obtained  by  simply  allocating  all  available  computer  time  to  replicating 
the  simulation  and  forgoing  the  use  of  the  ACV?  If  not,  what  value  of  G  provides  a  sufficiently 
accurate  estimate  of  /x^  while  allowing  enough  time  to  produce  a  sufficiently  narrow  CIW  for  the 
ACV-controlled  response?  The  answers  to  these  questions  depend  on  several  factors  that  are  un¬ 
known  before  the  simulation  is  replicated.  These  factors  include  the  times  required  to  produce  a 
simulation  replication,  a  single  ACV,  and  a  single  Monte  Carlo  estimate  of  /x-^.  Other  factors  to  be 
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considered  include  the  amount  of  variance  reduction  achieved  using  the  ACV,  and  the  comparative 
widths  of  the  confidence  intervals  about  the  ACV-controUed  response  and  Though  the  issues 
may  seem  complex,  sufficient  insight  can  be  obtained  in  a  straightforward  manner. 

To  determine  a  satisfactory  value  for  G,  begin  by  allocating  some  fi:action  of  the  available 
computer  time  to  perform  a  small  pilot  study  and  obtain  estimates  of  all  the  applicable  factors. 
Consider  the  case  where  a  fixed  amoimt  of  computer  time,  T,  is  allocated  for  a  complete  simula¬ 
tion  study.  The  goal  is  to  determine  a  satisfactory  value  for  G  that  is  both  accurate  and  efficient. 
To  accomplish  this,  compare  the  widths  of  the  predicted  confidence  intervals  for  the  imcontroUed 
and  ACV-controlled  responses,  based  on  the  pilot  study  values.  Label  the  times  required  to  pro¬ 
duce  a  simulation  replication,  and  ACV,  and  a  Monte  Carlo  replication  as  tREPy^ACV^  ^Jid  tMC 
respectively.  The  maximum  possible  number  of  uncontrolled  replications  is  given  by 


'TlUNC  — 


T 

^REP 


(4.6) 


where  [ij  returns  the  nearest  integer  less  than  or  equal  to  i.  The  predicted  (1  —  a)  100%  CIW  for 
the  uncontrolled  response  is  given  by 

(4.7) 

where  Sy  is  the  sample  variance  of  F.  To  predict  the  (1  —  a)100%  CIW  for  the  ACV-controlled 
response,  CIWacv^  we  select  an  initial  value  of  G.  The  maximum  possible  nximber  of  simulation 
replications  using  an  ACV  is  then  given  by 


CIWuNC  =  2ti_a/2,nt7Ar<7-l1 


iJy. 

nuNC 


nACV  — 


tRBP  +  ^ACV  +  GtMC 


(4.8) 


4r7 


An  initial  estimate  of  CIWjicv  obtained  using  Equation  (2.8)  to  estimate  Var  ^ 


om^cv  =  (1  -  --W  («) 

where  ry^  ^  mayirmun  likelihood  estimator  of  the  square  of  the  correlation  coefficient  between 
Y  and  Z  (the  ACV).  Then  adjust  iteratively,  re-computing  Equations  (4.8)  and  (4.9)  until  the 
CIW  about  fiz  is  approximately  10%  of  CIWacv^  Then  if  CIWacv  is  less  than  CIWunc^  fbe 
ACV-controUed  response  estimate  is  more  accurate  than  the  uncontrolled  estimate.  Otherwise  it 
is  more  efficient  to  forego  the  application  of  the  ACV,  Note  two  items.  First,  it  is  not  claimed 
that  Equation  (4.9)  is  an  exact  formula,  but  only  that  it  provides  a  useful  estimate.  Secondly,  it  is 
possible  to  arrive  at  a  value  of  G  that  is  both  efficient  and  provides  a  CIW  about  jXz  that  is  much 
narrower  than  10%  of  CIWacv^  It  is  left  to  the  best  judgement  of  the  simulation  practitioner  to 
determine  a  satisfactory  value  G  in  that  case. 

4,3  Queueing  Network  Example 

The  performance  of  the  various  control  variate  methods  can  be  compared  using  the  example 
queueing  network  example  shown  in  Figure  4.2.  Lavenberg,  Moeller,  and  Welch  [30]  have  previously 
examined  this  classic  model  within  a  variance  reduction  context.  The  queueing  network  is  composed 
of  S  service  stations  with  N  customers  circulating  between  them.  Call  this  model  Qi .  Station  1  has 
exactly  N  servers,  resulting  in  no  queueing  at  station  1.  The  remainder  of  the  stations  are  single 
server  queues,  all  employing  a  first  come-first  served  service  discipline.  The  transition  probability 
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matrix  for  customer  movement  between  stations  is  given  by 


0  1  0  0 

Pi  0  P3  •  • '  PS 

P=  0  1  0  0  (4-10) 

0  1  0  0 

with  ^f-iPi  =  !•  Any  customer  completing  service  at  station  A:  =  1,3, . ..  ,S  is  immediately 
routed  to  station  2.  Upon  completion  of  service  at  station  2,  the  customer  is  routed  to  station  k 
with  probability  pk-  Station  2  has  an  exponential  service  time  distribution  with  mean  A.  Each  of 
the  other  stations  has  a  Weibull  service  time  distribution  with  shape  parameter  a*  >  0  and  scale 
parameter  /3^  >  0.  The  mean  service  time  for  each  station  is  therrfore 

k  =  l,3,-,s  (4.11) 

and  the  variance  is 

„2  =  ^/«.jr(2^)-r(2^)'},  <=  =  i,3,....5  (4.12) 

where  r(*)  is  the  gamma  function. 

Model  Qi  is  a  simple  representation  of  an  interactive  multiprogrammed  computer  system.  The 
customers  in  the  network  are  users  of  the  system,  with  station  1  representing  the  user  terminals. 
Each  service  time  at  station  1  represents  a  user’s  “think”  time  between  system  task  requests. 
Station  2  represents  the  system’s  central  processing  unit  (CPU)  and  stations  3, . . .  ,5  denote  mass 
storage  units  (disk,  drum,  tape,  etc.).  The  service  time  at  station  2  represents  processing  time  imtil 
either  a  task  is  completed  (in  whidi  case  the  customer  returns  to  station  1)  or  data  from  a  mass 


storage  unit  is  required.  A  service  time  at  stations  3, . . .  ,5  represents  the  time  required  to  transfer 
data  from  the  storage  device  to  the  main  memory,  where  it  can  be  acted  upon  by  the  CPU. 

Users  must  be  allocated  a  portion  of  main  memory  in  order  to  aecess  the  CPU  and  storage 
devices.  Since  memory  is  limited,  all  users  may  not  be  permitted  memory  access  at  the  same 
time.  This  leads  to  a  variation  of  Qi  (called  Q2),  that  is  shown  in  Figure  4.3.  In  model  at 
most  N'  <  N  customers  can  enter  the  subnetwork  comprised  of  stations  2, . . .  ,  5.  A  new  queueing 
station  H  holds  customers  in  delay  until  the  customer  population  in  the  subnetwork  is  less  than 
AT'.  The  service  time  for  station  H  is  zero  and  no  queueing  occurs  at  H  if  the  customer  population 
in  the  subnetwork  is  less  than  AT'. 

Numerous  performance  measures  may  be  of  interest  for  these  notional  computer  systems, 
but  two  measures  are  particularly  important.  The  first  is  the  system  sojourn  time,  defined  here 
as  the  long-run  average  time  between  a  customer’s  visits  to  station  1.  The  other  measure  is  the 
steady-state  CPU  utilization  (the  long-run  fraction  of  time  station  2  is  busy  serving  customers). 
These  measures  respectively  address  effectiveness  and  eflSciency  and  are  of  interest  for  both  Q\ 
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Figure  4.3  Closed  queueing  network  Q2 


and  Q2-  We  estimate  these  steady-state  performance  measures  by  truncating  a  constant  number 
of  events  from  the  beginning  of  each  simulation  replication  to  eliminate  initial  transient  behavior 
bias.  The  reader  should  interpret  all  definitions  of  the  following  statistics  to  implicitly  include  these 
truncations. 

First  the  system  sojourn  time  is  estimated  in  the  following  manner.  Let  r  be  the  true  expected 
sojourn  time  that  we  wish  to  estimate,  and  let  Uj  be  the  realized  sojourn  time  during  replication 
j.  Defining  an  event  as  the  completion  of  service  at  any  of  the  network  stations,  each  replication 
will  be  terminated  upon  the  completion  of  M  events.  For  replication  define  rrij  as  the  number  of 
customer  returns  to  station  1  at  or  before  M  events.  Then  calculate,  the  sample  mean  sojourn 
time  for  replication  j  by 


TTlj 


1=1 


,n 


(4.13) 
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so  that  r  can  be  estimated  by 


f  =  Tin) 


(4.14) 


Let  V  be  the  true  steady-state  value  of  CPU  utilization,  and  let  bij  be  the  service  time  realized 
at  station  2  during  replication  j.  Note  that  this  can  be  assigned  to  any  of  the  customers  in  the 
system.  To  estimate  v,  define  ej  as  the  simulated  time  imtil  M  events  occur  and  qj  as  the  number 
of  service  completions  at  station  2  for  replication  j.  We  then  represent  the  CPU  utilization  sample 
mean  for  replication  by  {7j,  where 

=  j  =  (4.15) 

i=i 


so  that  V  can  be  estimated  as 


v  =  Uin)  =  ^'£Uj  (4.16) 

j-l 

4-3.1  Internal  Control  Variates.  As  described  in  the  chapter  II,  internal  control  variates 
are  the  random  variables,  or  functions  of  them,  generated  within  the  simulation  that  have  a  known 
mean.  Random  variables  meeting  these  criteria  for  models  Qi  and  Q2  include  the  5  service  times 
and  the  5  --  1  values  of  the  routing  proportions  for  stations  1,3, .. .  ,5.  Many  researchers  have 
explored  functions  of  the  input  random  variables  to  find  robust  and  asymptotically  stable  internal 
control  variates. 

Standardized  work  variables  are  chosen  for  the  S  service  time  ^^iables.  Wilson  and  Pritzker 
[56]  have  demonstrated  that  standardized  work  variables — standardized  statistics  of  the  service  time 
distribution  at  each  service  station — are  robust  and  asymptotically  stable  for  the  type  of  queueing 
system  studied  here.  The  standardized  work  variables  are  defined  this  way.  For  each  replication 
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j,  let  Sij{k)  represent  the  realized  service  time  at  stations  k  =  1,2,...  ,5.  Let  aj(k)  be  the 
total  number  of  service  completions  at  station  k  during  replication  j.  Then  the  standardized  work 
variables  Wkj  axe  given  by 


W  hi. 


k  —  1, 2, . . .  j  —  1,2,...  ,  n 


(4.17) 


where  and  ak  axe  given  in  Equations  (4.11)  and  4.12  respectively.  Then  eaeh  Wkj  has  a  mean 
of  zero  and  a  standard  deviation  of  one  as  aj{k)  — ^  oo  [56]. 


Given  the  candidate  control  variates  for  the  service  time  distributions,  the  focus  is  changed 
to  the  routing  random  variables.  The  choice  is  the  standardized  routing  variable — a  standardized 
statistic  of  the  routing  random  variables — has  been  shown  to  significantly  reduce  variance  based 
on  this  type  of  multinomial  construct  [10].  The  standardized  routing  variable  is  developed  in  the 
following  manner.  An  indicator  variable,/ij(fc)  is  defined,  such  that 


Iij{k)^ 


0 


if  the  station  2  departure  goes  to  station  k  for  replication  j 
otherwise 


(4.18) 


for  A:  =  1,3, . . .  ,5.  Here,  the  departure  refers  to  the  service  completion  at  station  2, 
regardless  of  the  customer.  Then  a  standardized  routing  variable  for  activity  k  is  defined  as 


aj{2) 

Rkj  =  X] 


Ijj  (A?)  Pk 


^  {(«i(2))(l-Pfe)Pfe} 


1/2- 


j  =  l,2,...  ,n;  A;  =  l,3,...  ,5 


(4.19) 


where  aj{2)  is  the  total  number  of  service  completions  at  station  2  during  replication  j.  Each  pk  is 
the  transition  probability  from  station  2  to  station  k  as  given  in  Equation  (4.10).  Bauer  and  Wilson 
have  shown  that  standardized  routing  variables  converge  to  a  normal  distribution  with  mean  zero, 
as  the  simulation  run  length  increases  [10]. 
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4.3.2  Analytical  Control  Variates.  To  find  ACV’s  for  models  Qi  and  Q2,  consider  a  new 
model  (called  Q3)  that  can  be  solved  analytically.  Let  Qs  have  the  same  structure  as  model  Qi, 
except  that  all  service  time  distributions  are  independent,  identically  distributed  (HD)  exponential 
random  variables  with  mean  service  times  equal  to  those  of  Qi.  Using  a  product-form  algorithm 
such  as  Mean  Value  Analysis  (MVA)  [13, 16,29],  the  steady-state  expected  sojourn  time  and  CPU 
utilization  for  Q3  can  be  determined  exactly.  As  discussed  in  chapter  11,  the  MVA  algorithm  is 
used  to  solve  queueing  networks  that  have  a  closed  form  solution.  An  MVA  approach  is  chosen 
since  it  is  easy  to  implement  and  reasonably  fast.  The  algorithm  yields  the  mean  values  of  response 
time,  queue  length,  throughput,  and  utilization  for  each  service  center  in  the  network.  It  solves 
the  network  by  first  determining  these  values  when  only  one  customer  is  in  the  network.  Using 
this  information,  the  network  is  solved  when  two  customer  are  in  the  system  using  the  mean  value 
theorem.  The  mean  value  theorem  relates  the  response  time  of  a  service  center  when  n  customers 
are  present  to  the  length  of  the  queue  when  n  —  1  customers  are  present.  The  algorithm  is  reappUed 
until  the  network  is  solved  for  the  total  number  of  customers  desired. 

Define  as  the  ACV  for  sojourn  time  and  C{U)f^^  as  the  ACV  for  CPU  utilization 

imder  replication  j.  For  each  ACV  calculate  the  sample  mean  service  times  and  the  realized 
branching  proportions  from  station  2  for  each  replication  and  then  input  them  into  the  MVA 
model  for  each  simulation  replication.  To  accomplish  this,  for  replication  j  define  Vij{k)  as  the 
realized  service  time  at  station  k.  Then  define  14j,  the  average  service  time  for  replication  j  at 
station  fc,  as 

aj{k) 

Vkj  = —7TT  T,  Vijik),  k=l,2,...,s-,j  =  l,2,...,n  (4.20) 


4-14 


where  aj{k)  is  the  number  of  service  completions  at  station  k  during  replication  j.  To  calculate 
the  realized  routing  proportions,  Pkj,  from  station  2  to  station  k  for  replication  let 


aj  (k) 

E  E 

k=l 

fc7^2 


k  —  1, 3, . . .  ,  S]  j  —  1, 2, . . .  ,71 


(4.21) 


where  the  indicator  variables,  lijik)^  are  as  defined  in  Equation  (4.18).  These  values  of  Vkj  and 
Pkj  are  then  input  into  the  MVA  algorithm  to  obtain  and  C{U)f^^  for  each  replication 

j- 

The  next  step  in  the  ACV  method  is  to  approximate  the  mean  of  the  ACV  (for  both  system 
sojourn  time  and  CPU  utilization)  using  the  Monte  Carlo  method  described  in  Section  4.2.2.  This 
method  cannot  be  directly  applied  since  the  random  variables  that  must  generated  are  not  inde¬ 
pendent.  The  routing  proportions  have  a  multinomial  distribution  and  are  therefore  correlated. 
Even  the  sample  mean  of  the  realized  service  times  are  correlated  to  the  routing  proportions,  since 
their  variance  is  a  function  of  the  number  of  times  customers  are  routed  to  their  respective  service 
centers. 

The  Monte  Carlo  method  is  modified  by  generating  correlated  random  variates  using  the 
conditional  distribution  method  described  in  Law  and  Kelton  [32].  The  method  requires  that 
the  complete  joint  distribution  of  the  random  variables  to  be  generated  be  known  as  well  as  the 
derivation  of  the  marginal  and  conditional  distributions.  The  method  begins  by  generating  a 
single  random  variate  from  its  marginal  distribution.  The  next  random  variate  is  generated  using 
its  marginal  distribution  conditioned  on  the  realization  of  the  first  random  variate.  The  scheme 
repeats  this  process  until  all  random  variates  are  generated.  In  this  particular  case,  the  required 
distributions  can  be  derived  since  the  routing  proportions  specify  a  multinomial  distribution. 


The  random  variates  generated  are  the  Monte  Carlo  routing  proportions,  and  the 

Monte  Carlo  service  times,  for  A:  =  1, 2, . . .  ,5  and  j  —  1, 2, . . . ,  G,  where  G  is  the  number 

of  Monte  Carlo  replications.  To  begin,  construct  the  multinomial  distribution  that  has  the  routing 
proportions  as  its  parameters.  The  multinomial  random  variables,  are  described  as  the  number 
of  times  customers  are  routed  from  station  2  to  station  k  for  Monte  Carlo  replication  j.  The  total 
number  of  routings  for  each  Monte  Carlo  replication  are  determined  based  on  the  simulation  model. 
Note  that  all  customers  return  to  station  2  following  service  at  any  other  station,  so  for  every  service 
completion  at  station  fe  ^  2,  there  is  a  paired  service  completion  at  station  2.  Since  the  total  number 
of  counted  events  for  every  simulation  replication  is  M,  the  total  number  of  routings  from  station 
2  to  the  other  service  stations  is  half  of  that,  or  M/2,  the  multinomial  distribution  of  customer 
routings  can  then  be  described 


{Akj,  fe  =  1, 3, . . .  ,  5}  ~  multmoroial  {[M/2]  ■,  Pi,P3,- ■  ■  ,Ps) ,  j  = 


(4.22) 


where  the  pk  are  as  described  in  Equation  (4.10).  Also,  A2j  =  M/2  for  all  j. 

The  following  is  a  scheme  to  generate  the  required  random  variates  using  the  conditional 
distribution  method  of  generating  correlated  random  variates  [32].  The  marginal  and  conditional 
distributions  are  derived  in  the  following  manner.  The  marginal  distribution  of  Akj  is  binomial  with 
parameters  [M/2]  andp/b(fc  ^  2).  The  marginal  distribution  of  conditional  on  the  realization 
of  Akj  =  akj ^  k  —  1,3, . . .  (^  <  5),  is  also  binomial  with  parameters  [M/2]  —  ^11=1  ^^3 

k^2 

Pq+i/  ^1  —  Using  the  normal  approximation  to  the  binomial,  generate  each  as 

follows: 


1.  Generate  Pij  ^  Normal 

2.  Let  Aij  =  Pii(M/2). 


3.  Generate  P^j  ~  Normal 


P3 


l-pi’  M/2-A^j 
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4.  Let  Asi  =  Psi  ((M/2)  -  Aij). 

5.  : 


6.  : 

i.  Generate  Ps-ij  ~  Normal 


Ps-\ 

1  * 

i+1.  Let  As-\j  —  Ps-ij 


s-i 


i+2.  Let  Asj  —  ^  ^  Akj  • 


fc=i 


i+3.  Let  Pif^  = 


k=l,3,...  ,S. 


(M/2) 

Given  these  realizations,  the  distribution  for  each  of  the  Monte  Carlo  generated  mean  service 
times  is  given  by 


Vkj^  ^  Normal  ,  fe  =  1, 2, . . .  ,  5  (4.23) 

where  A2j  =  M/2  and  known  mean  and  variance  of  each  service  activity.  Then 

for  each  Monte  Carlo  sample  j,  the  generated  values  of  the  routing  proportions  and  mean  service 
times  are  input  into  the  MVA  algorithm  to  obtain  '  The  mean  of  the  ACV  can  then  be 
approximated  by 

1=1 

4.3.3  External  Control  Variates.  External  control  variates  can  be  obtained  by  creating  a 
simulation  for  model  Qz-  The  true  steady-state  expected  response  time  (r®^^)  and  expected  CPU 
utilization  (u®-^^)  are  determined  by  the  MVA  algorithm.  Using  common  random  numbers,  this 
simulation  model  can  then  be  used  to  produce  external  control  variates  for  Q\  or  Qi.  However,  due 
to  the  nature  of  a  closed  network,  exact  synchronization  of  random  variates  becomes  very  difficult. 


4-17 


Law  and  Kdton  [32]  provide  an  excellent  discussion  of  common  random  numbers  and  the  problems 
associated  with  their  application.  For  these  models  some  common  random  number  synchronicity 
can  be  achieved.  Note  that  the  service  times  at  stations  1,3,...  ,  S'  in  or  Q2  are  distributed  as 
TTD  WeibuU  random  variables,  and  that  the  same  service  times  in  model  Qs  are  distributed  HD 
exponential.  The  inverse-transform  method  of  random  variate  generation  to  generate  the  service 
times  can  then  be  used  for  both  models  [32].  Thus  the  service  time  at  the  service  station 
is  generated  by  using  the  same  uniform  [0,1]  pseudo-random  number  for  both  models  Qi  (or  Q2) 
and  Q3.  Additionally,  the  same  uniform  [0,1]  pseudo-random  number  can  be  used  to  generate  the 
routing  random  variable  for  the  service  completion  at  station  2  for  both  models.  The  inability 
to  achieve  complete  synchronization  results  from  the  fact  that  the  service  time  at  station  k  will 
not  be  exactly  the  same  for  both  models  (albeit  highly  correlated).  This  situation  eventually  results 
in  different  sequences  (between  models)  of  specific  customers  arriving  at  any  particular  station. 

To  generate  the  external  control  variates  for  each  simulation  replication  j,  the  following  output 
statistics  for  simulation  model  Qs  are  calculated.  Note,  the  same  implicit  truncation  scheme  is  used 
for  Qs  as  it  is  for  Qi.  Let  be  the  realized  sojourn  time  during  replication  j.  As  before,  by 
defining  an  event  as  the  completion  of  service  at  any  of  the  network  stations,  each  replication  will 
be  terminated  upon  the  completion  of  M  events.  For  replication  j,  define  mf^'^  as  the  number  of 
customer  returns  to  station  1  at  or  before  M  events.  Then,  the  external  control  variate 

for  sojourn  time  for  replication  j,  is  calculated  by 


C{T)f^^  == 


m; 


EXT 


^EXT 

Hi 


3  =  1,2,, 


i=i 


(4.25) 


For  CPU  utilization,  let  be  the  service  time  realized  at  station  2  during  replication  j  of 
model  Qs.  Next  define  as  the  simulated  time  until  M  events  occur  and  as  the  number 
of  service  cc»npletions  at  station  2  for  replication  j.  Then  the  external  control  variate  for  CPU 
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utilization,  C{U)f  replication  j,  is  calculated  by 


C(U)f 


EXT  _ 


oEXT 


,n 


i=l 


(4.26) 


4-4  Performance  Comparison 

44  J  Experimental  Procedures.  Several  experiments  axe  conducted,  each  varying  in  the 
number  of  replications  and  network  parameters  chosen.  Twelve  different  network  design  points  (six 
for  Qi  and  six  for  Q2)  are  selected.  The  selection  of  particular  network  settings  is  discussed  below. 
At  each  design  point,  100  experiments  are  conducted  with  the  number  of  replications  for  each 
experiment  equal  to  10,  and  50  experiments  with  t^e  number  of  replications  for  each  experiment 
equal  to  20.  For  every  design  point,  performance  measures  for  internal,  analytical,  and  external 
control  variates  are  compared.  Estimated  variance,  confidence  interval  width,  coverage,  and  MSE 
values  for  the  controlled  responses  are  compared  to  the  same  values  for  the  uncontrolled  responses. 
For  the  internal  control  variates,  comparisons  are  made  for  all  possible  combinations  of  input 
variates  for  both  networks.  The  efficiency  of  the  ACV  controlled  responses  is  also  compared  with 
that  of  the  uncontrolled  responses. 

Comparisons  are  made  using  the  generalized  method  presented  by  Bauer  and  Wilson  [10].  Let 
be  the  expected  value  of  the  performance  measurement  of  concern.  For  the  n  =  10(20)  replications 
of  the  experiment,  ft  =  1, 2, . . .  ,  d  (d  —  100(50)),  an  estimate  of  /x  is  computed.  Call  the  estimate 
fikO),  where  I  =  1  and  I  —  2  denote  uncontrolled  and  controlled  estimates  respectively.  In  a  similar 
manner,  let  d|(Z),  Z  =  1, 2  denote  the  tmcontroUed  and  controlled  estimates  of  the  variance  of  Afe(0* 
Then  the  average  variance  estimator  over  all  d  experiments  for  a  given  setting  is 

i  =  (4.27) 
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The  percentage  change  in  estimated  variance  due  to  the  use  of  a  particular  control  variate  method 
is  then  estimated  by  100(d^  —  d^(l))/d^(l). 

For  the  experiment,  the  confidence  interval  estimate  is  given  by 

A{l)  =  fi,{l)±Hk{l)  (4.28) 


where  Hk{l)  is  the  estimated  half-width  as  given  in  Equation  (2.21)  with  a  =  0.10.  We  find  the 
average  width  of  the  confidence  interval  estimator  over  all  d  experiments  for  a  given  setting  as 


1  ^ 

2Hil)  =  1  =  1,2 


/l==l 


(4.29) 


Then,  as  with  the  variance  estimates,  the  percentage  change  in  the  estimated  confidence  interval 
width  due  to  the  use  of  a  particular  control  variate  method  is  estimated  by  100  ^H{2)  —  2ff  (1)^  /2H (1). 

An  important  concern  for  control  variate  performance  is  the  amoimt  of  bias  in  the  controlled 
estimate  of  //.  Bias  is  induced  because  P  must  be  estimated  and  is  generally  not  independent  of 
Y{n)  [30].  One  related  measure  of  bias  is  the  estimated  confidence  interval  coverage  probability. 

To  estimate  coverage,  let 


h{l)  =  { 


1 

0 


if  G  Afc(0 
otherwise 


(4.30) 


for  Z  =  1, 2  and  ft  =  1, 2, . . .  ,  d.  Then  an  estimate  of  the  confidence  interval  coverage  probability  is 
given  by  the  calculated  coverage  fraction  for  Afc(Z),  computed  as 


=  1  =  1,2 


(4.31) 


h=l 
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Realized  coverage  may  not  always  be  the  most  ioformative  indicator  of  bias.  For  example, 
a  point  estimate  may  be  very  close  to  /x,  but  if  the  associated  confidence  interval  is  small  enough, 
coverage  may  not  be  realized.  In  order  to  measure  bias  in  a  manner  that  considers  this  "closeness,” 
the  estimated  value  of  the  mean  square  error  (MSE)  of  a  point  estimator  is  computed  as 

-  m 

=  l  =  h2  (4.32) 

h=l 

The  true  expected  values  of  the  system  sojourn  time  and  CPU  utilization  are  estimated 
through  25,000  replications  at  each  design  pomt.  With  these  very  large  samples,  the  associated  .90 
confidence  intervals  axe  sufiiciently  tight  (less  than  0.5%  of  estimated  value  in  all  cases)  to  make 
good  benchmark  estimates  of  coverage  and  MSE  for  comparison  purposes. 

Another  concern  is  the  efficiency  of  the  ACV  method.  Generating  the  ACV  for  each  repli¬ 
cation  and  the  Monte  Carlo  replications  for  approximating  consume  available  computer  time. 
Depending  on  the  amount  of  variance  reduction  achieved,  a  smaller  confidence  interval  width  may 
be  achieved  by  simply  using  all  available  computer  time  to  generate  an  uncontrolled  response.  The 
widths  of  confidence  intervals  produced  by  an  ACV  controlled  response  using  20  simulation  repli¬ 
cations  are  compared  with  that  predicted  for  an  uncontrolled  response  over  an  equivalent  amount 
of  time.  As  in  section  4.2.3,  we  let  tuEPjtACV,  and  Imc  represent  the  times  required  to  generate 
a  single  replication  of  Qi{Q2)i  a  single  ACV,  and  a  single  replication  of  the  Monte  Carlo  approxi¬ 
mation  of  respectively.  Then  the  time  required  to  produce  an  ACV  controlled  response  for  20 
simulation  replications  is  determined  as 

Tacv  =  20  {tREP  +  tAcv)  +  GtMC  (4.33) 
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The  equivalent  number  of  replications  possible,  if  allocated  all  of  Tacv  is  allocated  to  replications 
of  Qi{Q2)  only,  is  given  by 


'n^REP  = 


Tacv 

tREP 


(4.34) 


where  [ij  returns  the  closest  integer  less  than  or  equal  to  i.  The  estimated  equivalent  confidence 
interval  width  is 

(4.35) 

where  Sy  is  the  sample  variance  calculated  over  all  1,000  replications  at  the  appropriate  design 
point.  CIWeqv  is  compared  to  the  average  confidence  interval  width  achieved  by  the  ACV  con¬ 
trolled  response  at  each  design  point. 

ACV  eflSciency  is  also  examined  by  comparing  the  times  required  to  produce  an  equivalent  size 
confidence  interval  about  an  uncontrolled  and  ACV  controlled  response.  The  average  confidence 
interval  widths  of  the  ACV  controlled  response  using  20  replications  are  used  as  the  base  line.  Then 
the  predicted  number  of  replications  required  to  achieve  an  equivalent  confidence  interval  width 
about  an  uncontrolled  response  is  determined  by 

(4.36) 

where  Sy  is  the  same  as  above  and  CIWacv  is  the  average  confidence  interval  width  about  the 
ACV  controlled  response.  The  time  required  to  complete  ueqv  replications  of  model  Qi(Q2)j  given 
by  TiEQvtREPj  is  then  compared  to  the  time  required  to  produce  an  ACV  controlled  response  of 
20  rephcations,  given  by  20  {tREp  +  ^acv)  +  GImc- 


CIWeqv  =  '^h-a/2 


I  SI 

riREP 
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Network  Settings,  For  the  closed  queueing  network  presented  in  Section  4.3,  six 
different  experimental  design  points  axe  selected  for  both  models  Qi  and  (52-  For  all  experiments, 
the  number  of  service  stations  (S)  is  6  and  the  number  of  customers  ( JV)  is  25.  For  model  Q2 ,  the 
number  of  customers  allowed  into  the  constrained  subnetwork  (N^)  is  5  for  all  settings.  To  create 
the  six  design  points  for  each  model,  two  different  transition  probability  matrices  are  applied  to 
three  sets  of  service  time  distributions.  The  two  transition  probability  distributions  are  provided 
in  Table  4,1  and  the  three  service  time  settings  are  listed  in  Table  4.2.  These  settings  are  created 
to  stress  the  queueing  network  at  different  stations  to  determine  the  effectiveness  of  each  type  of 
control  variate  under  various  network  flow  conditions.  Results  are  obtained  for  each  design  point 
using  replication  sizes  of  both  10  and  20. 


Table  4.1  Transition  probability  matrix  values. 


Matrix 

Pi 

P3 

P4 

P6 

Pe 

Pi 

0.20 

0.36 

0.36 

0.040 

0.040 

P2 

0.25 

0.30 

0.30 

0.075 

0.075 

In  all  twelve  cases,  replications  of  the  simulation  are  terminated  foUowing  the  completion  of 
2,000  events.  To  remove  the  effect  of  the  ioitial  transient  behavior,  data  from  the  first  500  events  is 
ignored.  In  addition,  the  initial  state  of  the  network  (number  of  customers  at  each  station)  is  based 
on  the  expected  number  of  customers  at  each  station  for  analytical  model  Q3.  These  expected 
values  are  determined  by  solving  the  system  using  Mean  Value  Analysis.  The  approximate  steady- 
state  probability  that  a  customer  is  at  a  given  station  is  determined  by  dividing  the  expected 
number  of  customers  at  a  service  station  by  the  number  of  total  customers.  Then,  at  the  start  of 
each  rephcation,  each  customer  is  assigned  a  uniform  [0,1]  pseudo-random  number  and  is  routed 
to  a  particular  station  by  corresponding  probabilities.  (?,  the  number  of  Monte  Carlo  replications 
required  to  approximate  is  set  to  equal  10,000  at  all  design  points.  The  width  of  the  90% 

confidence  interval  about  is  approximately  10%  of  the  width  of  the  90%  confidence  intervals 
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Table  4.2  Service  time  distribution  settings. 


Service  Center 

Distribution 

a 

P 

Mean 

Variance 

Setting  A 

1 

1.46824 

2 

r>~> 

mm 

hhq&i 

3 

5.64760 

10.0 

0.08 

4 

5.64760 

10.0 

■013 

0.08 

5 

WeibuU 

2.61249 

26.44 

6 

WeibuU 

2.61249 

1000.0 

■MMil 

26.44 

Setting  B 

1 

1.46824 

1000.0 

4795.78 

2 

r\j 

1.00 

3 

10.0 

mm 

4 

10.0 

5 

2.61249 

1000.0 

26.44 

6 

2.61249 

1000.0 

26.44 

Setting  C 

1 

4795.78 

2 

exponential 

1 

1.00 

3 

WeibuU 

10.0 

1.39 

4 

WeibuU 

mm 

KISgl 

0.08 

5 

WeibuU 

6 

WeibuU 

2.61249 

1000.0 

12.50 

26.44 

about  the  ACV  controlled  responses  at  all  design  points  when  G  =  10, 000.  This  value  of  G  also 
worked  well  in  terms  of  ACV  ejSiciency. 


4^4-3  Results,  ACV’s  provide  significant  confidence  interval  reduction  on  estimates  for 
both  system  sojourn  time  and  CPU  utilization  for  the  closed  queueing  network.  Across  the  range 
of  all  experiments,  ACV  performance  is  typically  similar  to  that  of  external  and  internal  methods. 
Confidence  interval  width  reductions,  as  a  percentage  of  the  uncontrolled  estimated  confidence 
interval,  are  provided  in  Tables  4.3  and  4.4.  Results  are  included  for  ACV’s,  external  control 
variates,  and  internal  control  variates.  The  internal  case  represents  the  combination  of  standardized 
work  variables  and  standardized  routing  variables  that  produces  the  greatest  reduction  in  confidence 
interval  width.  Due  to  the  similarity  in  variance  reduction  achieved,  only  results  for  20  replications 
are  provided. 
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Table  4.3  Confidence  interval  width  reduction  (System  sojourn  time) 


System  sojourn  time 

Model 

Service  time 
setting 

Transition 

probability 

matrix 

Confidence  interval  wic 

th  reduction  (%) 

Analytical 

control 

variate 

External 

control 

variate 

Internal 

control 

variates 

Qi 

A 

Pi 

53.7 

52.5 

56.9 

P2 

46.0 

II^QI 

B 

Pi 

47.5 

66.4 

P2 

49.9 

52.4 

C 

Pi 

28.5 

43.6 

P2 

64.3 

O2 

A 

Pi 

49.2 

P2 

46.0 

B 

Pi 

62.3 

39.8 

68.0 

P2 

46.8 

C 

Pi 

24.4 

49.7 

P2 

58.2 

10.6 

68.6 

Of  particular  interest  is  the  performance  of  external  and  analytical  methods  for  model  Q2 .  The 
external  method  fails  to  provide  the  same  level  of  confidence  interval  width  reduction  (particularly 
for  CPU  utilization)  for  model  Q2  as  for  Qi.  Recall  that  only  five  customers  at  a  time  are  allocated 
a  portion  of  main  memory,  whereas  model  Qi  has  an  tmconstrained  subnetwork  and  an  unlimited 
number  of  customers  may  access  the  CPU  and  storage  devices.  The  variance  in  external  control 
variate  performance  is  probably  due  to  this  dissimilarity  in  model  structure.  For  model  Q2,  the 
common  random  numbers  lose  more  synchronization  and  the  system  responses  are  not  as  highly 
correlated.  ACV’s,  on  the  other  hand,  continue  to  perform  at  about  the  same  level  for  Q2  as  for 
Qi.  Although  the  underlying  analytical  model  is  also  unconstrained,  the  model  relies  only  on  the 
mean  responses  of  system  parameters  from  Q2-  Therefore,  the  same  conditions  that  cause  longer 
sojourn  times  or  greater  CPU  utilization  levels  in  model  Q2  will  do  so  in  the  analytical  model. 
Hence,  correlation  from  replication  to  replication  is  maintained  and  ACV’s  perform  well  for  Q2- 

Realized  coverage  and  estimated  MSE  estimates  for  uncontrolled  and  controlled  responses  are 
enumerated  in  Tables  4.5  and  4.6.  Nominal  coverage  is  90%.  Due  to  the  similarity  in  the  results, 
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Table  4.4  Confidence  interval  width  reduction.  (CPU  utilization) 


CPU  utilization 

Model 

Service  time 
setting 

Transition 

probability 

matrix 

Confidence  interval  wic 

Ith  reduction  (%) 

Analyiiical 

control 

variate 

Bxtemal 

control 

variate 

Internal 

control 

variates 

Qi 

A 

Pi 

55.4 

46.8 

47.1 

P2 

60.5 

50.1 

54.3 

B 

Pi 

70.6 

81.1 

P2 

57.9 

47.8 

C 

Pi 

42.7 

55.8 

34.8 

P2 

83.6 

Q2 

A 

Pi 

38.3 

34.9 

P2 

43.7 

30.9 

B 

Pi 

65.4 

65.5 

P2 

3.6 

32.4 

C 

Pi 

44.9 

10.5 

42.3 

P2 

69.4 

9.7 

71.3 

only  the  statistics  for  20  replications  are  provided.  Internal  control  variate  results  correspond  to  the 
same  internal  control  variate  combinations  reported  above  for  confidence  interval  width  reduction. 

The  approximation  of  /i  using  the  Monte  Carlo  technique  has  not  induced  any  detectable 
bias  in  the  analytically  controlled  estimates  of  sojourn  time  and  CPU  utilization.  Both  realized 
coverage  percentages  and  estimated  MSB’s  are  similar  for  all  three  t3rpes  of  control  variates,  with 
no  indications  of  any  significant  bias.  Additionally,  the  estimated  MSB’s  for  all  controlled  responses 
are  smaller  than  those  of  the  uncontrolled  estimates. 


Figures  4.4  and  4.5  illustrate  the  performance  of  the  ACV  compared  to  the  uncontrolled 
response  and  the  internal  and  external  controlled  responses.  The  figures  also  depict  the  associated 
confidence  intervals  for  50  experiments  at  a  particular  network  setting  and  20  replications  per 
experiment.  For  system  sojourn  time,  model  Q1  with  service  time  setting  B,  transition  probability 
matrix  P2,  and  20  replications  (50  experiments)  is  presented.  The  50  diamond  symbols  in  each 
figure  represent  the  50  point  estimates.  The  bracketed  lines  above  and  below  each  diamond  represent 
the  width  of  the  estimated  confidence  interval  for  the  associated  20  replication  design  point.  For 
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Table  4.5  Realized  coverage  (nominal  =  90%)  and  estimated  MSE.  (System  sojourn  time) 


System  sojourn  time 

Model 

Service 

time 

setting 

Transition 

probability 

matrix 

Coverage  percentage 

[Estimated  MSE) 

Uncontrolled 

response 

Analytical 

control 

variate 

Ektemal 

control 

variate 

Internal 

control 

variates 
^ - 

Qi 

m 

Pi 

92  (3.15) 

96  (0.61) 

90  (0.71) 

P2 

82  (0.69) 

B 

Pi 

86  (22.53) 

P2 

84  (5.24) 

94  (0.84) 

C 

Pi 

82  (1.47) 

P2 

88  (38.64) 

88  (19.53) 

Q2 

A 

Pi 

90  (5.83) 

98  (0.72) 

P2 

K3KE9I 

msw 

B 

Pi 

P2 

96  (1.43) 

C 

Pi 

P2 

88  (54.50) 

90  (8.13) 

88  (4.26) 

reference,  the  estimated  mean  found  using  25,000  replications  is  represented  by  the  horizontal  line 
across  each  figure.  For  CPU  utilization,  model  Q2  with  service  time  setting  C,  probability  transition 
matrix  P2,  and  20  replications  is  used  to  create  Figure  4.5. 

The  figures  not  only  illustrate  the  confidence  interval  width  reduction  achieved  when  us¬ 
ing  control  variate  methods,  but  also  illustrate  the  improved  accuracy  of  the  point  estimates  (or 
consistently  lower  values  for  estimated  MSE)  for  the  controlled  responses  when  compared  to  the 
imcontroUed  responses.  Additionally,  bias  does  not  appear  to  be  a  problem  with  any  of  the  control 
variates  illustrated,  even  for  the  ACV’s  using  Monte  Carlo  approximations  of  fi^- 

ACV  efficiency  comparisons  are  provided  in  Tables  4.7  and  4.8  for  both  time  equivalent  and 
confidence  width  equivalent  results.  All  comparisons  are  based  on  ACV  confidence  intervals  pro¬ 
duced  using  20  replications  of  the  simulation  model.  All  computer  generation  times  are  calculated 
using  our  results  on  a  Sun  SPARC  2  workstation.  The  computer  times  are  tREP  =  l.Ssec,  Iacv  = 
O.Olsec,  ndtMC  =  O.OOSsec,  with  Tacv  =  66.2sec  and  urep  =  36  replications  for  the  time  equiv¬ 
alent  comparisons.  The  comparison  ratios  indicate  the  relative  efficiency  of  the  ACV  by  comparing 
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Table  4,6  Realized  coverage  (nominal  =  90%)  and  estimated  MSE.  (CPU  utilization) 


System  sojovim  time 

Service 

time 

setting 

Transition 

probability 

matrix 

Coverage  percentage 

Estimated  MSE) 

Uncontrolled 

response 

Anal3d;ical 

control 

variate 

External 

control 

variate 

Internal 

control 

variates 

Qi 

A 

Pi 

86  (1.83) 

92  (2.36) 

P2 

88  (1.47) 

84  (2.88) 

88  (2.59) 

B 

Pi 

80  (1.77) 

92  (0.48) 

P2 

98  (0.82) 

90  (1.49) 

C 

Pi 

MUM  mill 

P2 

■awfcwaii 

^0^ 

Qi 

A 

Pi 

■iiiwfeiagii 

P2 

^^1CIE£)1 

B 

Pi 

^(!U!» 

P2 

96  (1.68) 

88  (4.81) 

C 

Pi 

90  (18.10) 

94  (4.24) 

88  (14.70) 

P2 

90  (18.60) 

92  (1.99) 

92  (1.77) 

the  time  required  to  produce  a  20  replication  ACV  controlled  response  to  the  time  equivalent  and 
confidence  mterval  width  equivalent  times  of  the  imcontroUed  response.  Values  greater  than  1  indi¬ 
cate  that  the  ACV  controlled  response  is  more  efficient  than  the  imcontroUed  response;  values  less 
than  1  indicate  that  the  equivalent  uncontrolled  response  is  more  efficient  than  the  ACV  controlled 
response. 

The  results  indicate  that  the  ACV  method  is  more  efficient  for  all  performance  measures  at  all 
design  points  except  for  one.  Even  in  that  case,  the  comparison  ratios  are  nearly  equal  to  1.  Given 
that  the  coverage  and  MSE  estimates  for  the  ACV  method  indicate  no  detectable  bias  the  10% 
guideline  for  the  confidence  interval  width  to  that  of  the  ACV  controlled  response  confidence 
interval  width  appears  to  be  appropriate.  Further,  this  choice  of  G  has  provided  an  ACV  controlled 
response  that  is  efficient  as  well. 

Although  there  are  12  different  design  points,  only  6  different  Monte  Carlo  approximations 
of  be  calculated  since  the  same  analytical  model  is  used  for  both  Qi  and  Qs.  In  order 

to  be  fair  the  efficiency  comparisons  are  made  as  if  is  calculated  at  every  design  point,  when 
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UncontroDed  Response 


Analytical^  CmitroHed  Response 


Eiq)ermients 

ExteraaDy  Controlled  Response 


I  50  H 


E^qieriments 


0  Estimated  sojourn  time 
I  Confidence  interval 
- Mean  =  46.3 


Figure  4.4  Experimental  results  with  estimated  confidence  interval.  Design  point:  Qi,  service 
time  setting  B,  transition  matrix  P2. 


in  actuality  the  “true”  efficiency  is  twice  that  reported.  Depending  on  the  type  of  experimental 
design  required  for  a  particular  simulation  study,  this  same  situation  can  occur,  further  increasing 
the  efficiency  of  the  ACV  controlled  response. 


4,5  Conclusion 

This  chapter  demonstrates  that  a  hybrid  type  of  control  variate,  called  an  ACV,  can  effec¬ 
tively  reduce  point  estimate  variance  from  replicative  simulation  studies  while  avoiding  some  of 
the  technical  difficulties  of  internal  and  external  control  variates.  In  terms  of  confidence  interval 
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Figure  4.5  Experimental  results  with  estimated  confidence  interval.  Design  point:  Q2j  service 
time  setting  C,  transition  matrix  P2. 


width  reduction,  the  experimental  results  indicate  that  ACV’s  are  quite  successful  for  the  networks 
studied.  This  supports  the  findings  of  previous  researchers. 

Previous  researchers  also  reported  unacceptable  levels  of  bias  for  point  estimates  foimd  using 
ACV’s.  This  bias  is  caused  by  the  use  of  inaccurate  values  for  the  mean  of  the  analjrtical  model 
when  given  inputs  from  the  simulation  model.  This  research  presents  a  simple  Monte  Carlo  method 
for  appra?dmating  the  mean  of  the  analytical  model  that  eliminates  detectable  bias. 

The  Monte  Carlo  method  should  be  general  enough  to  be  used  for  many  discrete  event  simu¬ 
lation  model  where  the  distribution  of  the  observed  sample  means  of  the  simulation  input  random 
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Table  4.7  Efficiency  comparisons.  (System  sojourn  time) 


System  sojourn  time 


Service 

time 

setting 

^^9 

Time  Equivalent 

CIW  Equivalent 

m 

Efficiency 

ratio 

UNC 

Rep.’s 

Efficiency 

ratio 

Qi 

A 

Pi 

1.60 

89 

P2 

KSl 

1.39 

68 

1.85 

B 

Pi 

1.41 

70 

P2 

KSil 

1.49 

78 

C 

Pi 

5.94 

1^ 

1.04 

39 

P2 

14.33 

13.72 

0.96 

34 

Q2 

A 

Pi 

1.76 

mm 

P2 

3.59 

1.63 

2.54 

B 

Pi 

5.91 

1.97 

\  134 

3.65 

P2 

4.23 

Hlsl 

3.08 

c 

Pi 

6.86 

10.55 

1.54 

■1^ 

2.26 

P2 

8.77 

15.65 

1.78 

111 

3.03 

variables  is  known  either  exactly  or  approximately.  The  use  of  the  conditional  distribution  method 
for  generating  correlated  random  variates  given  a  multinomial  distribution  should  be  applicable 
to  almost  any  discrete  event  simulation  model  that  contains  probabilistic  branching.  However, 
for  many  simulation  models  the  distribution  of  the  observed  sample  means  of  some  or  aU  of  the 
simulation  input  random  variables  may  not  be  known  either  exactly  or  approximately.  In  those 
cases  a  re-sampling  (bootstrap)  method  or  a  combination  re-sampling  and  Monte  Carlo  method 
could  be  used  to  approximate  the  mean  of  the  analytical  model.  Further  research  into  these  types 
of  models  to  determine  the  accuracy  and  efficiency  of  re-sampling  or  a  combination  re-sampling 
and  Monte  Carlo  method  could  increase  the  number  of  simulation  models  where  an  ACV  could  be 
used.  Exactly  this  type  of  research  is  conducted  in  the  next  chapter. 

Another  possible  problem  for  the  Monte  Carlo  method  could  occur  if  very  few  observations 
are  obtained  for  a  particular  input  over  the  course  of  a  replication.  Under  these  conditions,  the 
central  limit  theorem  effect  may  not  be  powerful  enough  and  a  normal  distribution  assumption 
may  be  inappropriate.  This  may  or  may  not  be  a  problem  though.  A  small  number  of  observations 
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Table  4.8  Efficiency  comparisons,  (CPU  utilization) 


System  sojourn  time 


Model 

Service 

time 

setting 

Transition 

probability 

matrix 

^^9 

Time  Equivalent 

CIW  Equivalent 

UNO 

CIW 

Efficiency 

ratio 

UNO 

Rep.’s 

Efficiency 

ratio 

Qi 

A 

Pi 

mamm 

1.64 

94 

2.56 

P2 

■iiiyAfci 

1.87 

121 

3.30 

B 

Pi 

0.0101 

0.0252 

2.51 

216 

P2 

B2!U£II 

1.76 

Pi 

■iiifeeiMi 

1  0.0319 

1.29 

HU 

1.61 

P2 

0.0270 

1.35 

64 

1.75 

Q2 

Pi 

mmum 

1.19 

51 

1.39 

P2 

1.32 

62 

1.69 

B 

Pi 

WmmM 

2.14 

4.34 

P2 

HDQHI 

44 

C 

Pi 

mmmm 

■iiis«Ka 

■KHil 

63 

1.72 

P2 

0.0143 

0.0343 

2.41 

200 

5.45 

for  a  particular  input  could  indicate  that  the  activity  it  represents  has  little  effect  on  the  overall 
performance  of  the  system  under  study.  Hence,  an  inappropriate  approximation  of  its  distribution 
could  have  little  effect  on  the  accuracy  of  our  approximation  of 

Although  the  use  of  a  Monte  Carlo  approach  to  approximating  the  mean  of  the  analytical 
model  requires  some  effort  and  computer  time,  the  extra  effort  can  pay  off  given  sufficient  variance 
reduction  is  achieved  by  the  ACV.  The  required  programming  is  very  simple  and  the  computer  time 
required  to  generate  Monte  Carlo  observations  is  very  small.  The  results  of  this  chapter  indicate 
that  the  Monte  Carlo  method  can  be  an  efficient  means  of  applying  an  ACV. 
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V.  ACV  Monte  Carlo  Method  with  Incomplete  Distribution  Knowledge 


5.1  Overview 

In  the  previous  chapter,  it  was  shown  that  the  Monte  Carlo  method  for  approximating  the 
mean  of  the  ACV  is  an  eflEective  means  of  resolving  the  ACV  bias  problem.  In  order  to  use  the 
Monte  Carlo  method,  the  means  and  variances  of  each  of  the  random  variables  used  as  inputs 
to  the  analytical  model  must  be  known.  In  addition,  the  probability  distribution,  including  the 
dependence  relationships,  of  all  the  input  variables  must  be  known  either  exactly  or  approximately. 
In  general,  these  conditions  will  not  always  be  met.  Depending  on  the  underlying  system  and  the 
way  a  simulation  model  of  the  system  is  constructed,  the  means,  variances,  or  dependence  structure 
of  some  or  all  of  the  inputs  to  the  analytical  model  may  not  be  known.  In  fact,  a  simulation  model, 
vice  an  analytical  model,  may  be  constructed  for  exactly  those  reasons.  Rule  based  routings  of 
customers  or  resources  is  just  one  example.  In  this  chapter,  methods  for  approximating  the  ACV 
mean  when  some  or  all  of  these  conditions  are  not  met  are  explored. 

The  chapter  begins  with  descriptions  of  different  methods  of  generating  random  vectors. 
Non-parametric  methods  based  on  an  observed  random  sample  are  described  first,  followed  by  a 
discussion  on  parametric  methods  of  generating  random  vectors.  These  methods  are  tested  on  a 
simulation  model  based  on  the  AMC  BRACE  airfield  simulation  model.  This  is  an  excellent  model 
to  use,  since  some  of  the  sample  means  produced  by  the  model,  necessary  to  generate  an  ACV, 
have  imknown  means  and  variances.  Both  the  simulation  model  and  the  analytical  model  used 
to  produce  an  ACV  are  described  in  the  following  section.  Experimental  methods  and  results  are 
presented  in  the  final  sections  of  this  chapter. 

5.2  Non-parametric  Approximation  Methods 

The  Monte  Carlo  method  described  in  Chapter  IV  approximates  by  generating  random 
variate  vectors  based  on  a  parametric  distribution  approximation  of  the  input  to  the  analytical 
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model,  Y^.  There  exist  non-paxametric  methods  that  don’t  require  any  assumptions  about  the 
underlying  distribution  to  generate  additional  random  vectors  based  on  the  observed  data.  These 
methods  rely  on  some  form  of  re-sampling  of  the  observed  data.  The  two  re-sampling  methods 
explored  are  the  bootstrap  and  SIMDAT  methods. 

5.2 J  Bootstrap.  One  approach  that  doesn’t  require  any  assumptions  about  the  distribu¬ 
tion  of  the  input  to  the  analytical  model  or  any  explicit  knowledge  of  the  correlation  structure  is 
the  bootstrap  re-sampling  technique.  Efron  first  described  the  bootstrap  in  an  attempt  to  better 
understand  the  jackknife  estimator  of  the  standard  deviation  of  a  distribution  [20]  and  has  since 
been  applied  to  numerous  statistical  problems.  The  bootstrap  is  described  in  the  following  man¬ 
ner  [21].  Consider  a  random  variable  ii(X,  F)  where  X  =  (Xi, X2, . . .  ,  X„)'  is  a  random  sample 
that  is  IID  distributed  from  distribution  F.  Then  the  bootstrap  estimate  of  E[R]  designated  by 
E*[R*]  is  found  in  the  following  manner: 

1.  Form  the  non-parametric  MLE  of  F  (also  referred  to  as  the  empirical  distribution) 


F:  mass  1/n  at  ar^,  i  =  l,  2,  ...,n 


2.  Draw  a  ‘T^ootstrap  sample”  (with  replacement)  from  F 


(5.1) 


(5.2) 


and  calculate  R*  =  i?(X*  ,F). 

3.  Independently  repeat  step  2  a  large  number  of  B  times,  obtaining  “bootstrap  replications”: 
Rl,R^, . . .  ,  and  calculate 

6=1 
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Since  R  is  any  random  variable  based  on  a  sample  from  a  parent  distribution,  the  bootstrap 
technique  could  be  used  to  estimate  the  mean  of  the  ACV  given  by  =  /j>z-  If 

Cov  )  =  0  for  *  =  1, 2, . . .  ,  g;  i  =  1, 2, . . .  ,  g;  i  #  i  the  joint  empirical  distribution  of 

the  observed  values  over  n  replications  of  =  (Y^iij)  ’  ^(2j) >  •  •  •  >  ^(gj)^  for  j  =  1, 2, . . .  ,  n,  is 
given  by 

F  :  ^mass  1/n  at  mass  1/n  at  %•, . . .  ,mass  1/n  at  YAiqj)^  (5-4) 

for  j  =  1, 2, . . .  ,  n.  Hence  there  are  different  vectors  that  can  be  sampled.  A  bootstrap  estimate 

of  can  be  found  by  first  drawing  a  bootstrap  sample  from  F 

with  Y^;^^.)  =  •  •  •  >  ^i(gi))'  for  J  =  1, 2, . . .  ,  n  and  calculate  by 

=  (5.6) 

^  -  1 
J=1 

Then  E[f^{<l)^^  Y^)]  is  estimated  by  repeating  the  steps  described  by  Equations  (5.5)  and  (5.6)  B 
times  and  finding 

E*\f^* 

However,  if  the  elements  of  Y^(i)  are  not  independent,  the  empirical  distribution  is  defined  by 

F  :  mass  1/n  atY^^^^  j  =  1, 2, . . .  ,  n  (5.8) 


1  ^ 

=  rE/" 


Any  resulting  bootstrap  sample  will  re-sample  only  the  original  n  data  points,  offering  little  gain 
in  accuracy  over  a  sample  mean  of  the  ti  observed  values  of  Z,  Therefore,  the  bootstrap  is  best 
suited  for  situations  where  the  random  variates  are  independent,  or  nearly  so. 


5.2.2  SIMDAT.  Taylor  and  Thompson  [50]  developed  an  algorithm  for  generating  random 
vectors  based  on  the  observed  values  of  a  multivariate  random  vector  X.  The  method  is  referred 
to  as  the  SIMDAT  method.  The  algorithm  generates  “pseudo-data  points”  that  behave  as  though 
they  come  from  the  underlying  distribution  of  X  without  knowing  or  estimating  the  underlying 
distribution.  Instead,  the  observations  are  combined  using  stochastic  multipliers  to  generate  the 
pseudo-observations. 

The  SIMDAT  algorithm  is  described  in  the  following  manner  [52].  Assume  the  goal  is  to  gen¬ 
erate  pseudo-random  data  points  from  the  underlying,  imknown,  distribution  of  a  random  sample 
where  Xj  =  a:2j, . . .  ,  XkjY .  The  first  step  is  to  standardize  the  sample  points  so  that 

the  marginal  sample  variance  of  each  random  variable  is  roughly  the  same.  For  some  given  integer 
m,  find  the  m  —  1  nearest  neighbors  of  each  of  the  n  different  random  samples  and  store  the  results 
in  an  array  of  size  n  x  (m  —  1). 

The  goal  is  to  generate  a  pseudo-sample  of  size  N.  Unlike  the  bootstrap,  there  is  no  need  for 
N  to  equal  n.  Now  select  one  of  the  n  observed  data  points  with  probability  1/n  and  retrieve  it’s 
m  —  1  nearest  neighbors  in  order  to  calculate  the  mean  of  the  resulting  m  data  points  by 


m  w  m  -  m  \  ' 

J 


Next,  code  the  selected  m  data  points  about  X  as 


(5.9) 


{xf}  =  {x,-x}7^, 


(5.10) 
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Although  these  computations  of  the  sample  means  and  coded  variables  are  presented  as  part 
of  the  simulation  process,  these  computations  need  only  be  done  once  before  the  simulation  process 
be^ns.  As  with  the  m  —  1  nearest  data  points,  the  resulting  and  X  values  can  be  stored  in  an 
array  of  size  n  x  (m  + 1)  for  later  recall. 

The  next  step  is  to  generate  m  random  variables,  ,  1^2 ,  •  *  •  ,  from  the  univariate  uniform 

distribution  defined  by 


U 


m 


/3(m-l) 

^  m  \ 

1  rm? 

(5.11) 


These  random  variables  are  then  used  to  generate  a  centered  pseudo-data  point  X^  by 


=  (5.12) 

1=1 

The  pseudo-data  point  Xp  is  obtained  by  adding  back  X  to  the  centered  pseudo-data  point 


Xp  =  X^  +  X 


(5.13) 


These  procedures  are  then  repeated  AT  times  to  generate  the  required  pseudo-data  points. 

The  algorithm  is  motivated  in  the  following  manner  [50].  Consider  a  sampled  vector  Xi  and 
its  m  —  1  nearest  neighbors 


(5.14) 

Assume  that  the  observed  data  points  are  from  a  truncated  distribution  with  mean  vector  //  and 
covariance  matrix  S.  Further  let  {uiY^^  be  an  independent  sample  from  the  uniform  distribution 
defined  in  (5.11)  above.  Therefore  E\ai]  =  1/m,  Var\ai\  =  (m  —  l)/m^,  and  Cov[ai^Uj]  =  0,  for 
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By  forming  the  linear  transformation 


m 

Z  =  ^uiXi  (5.15) 

the  component  of  Z  is  Zr  —  uiXri  -\-U2Xr2'i - \-Um^rm  resulting  in  the  following  relationships: 


E[Zr]  =  llr 

(5.16) 

Var[zr]  =  cl  +  {{m-l)/m}nl 

(5.17) 

Cov[Zr,  z^]  =  <Jrs  +  {{m  -  l)/m} 

(5.18) 

Observe  that  the  linear  transformation  results  in  Z’s  that  are  uncorrelated.  If  the  mean  vector 
of  X  is  /X  =  (0, 0, . . .  5  0)',  the  mean  vector  and  covariance  matrix  of  Z  is  identical  to  that  of  X  since 
E[zr]  =  0,  Var[zr]  =  and  Cov[zr,  Zs]  =  (Trs-  For  the  SIMDAT  algorithm,  the  translation  to  the 
local  mean  of  the  nearest  neighbor  cloud  wiU  not  achieve  these  results  exactly.  However  it  is  argued 
that  the  SIMDAT  algorithm  generates  points  having  very  nearly  the  same  mean  and  covariance 
structure  as  the  underlying  distribution  of  the  points  in  the  nearest  neighbor  cloud  [50,52]. 

The  selection  of  the  appropriate  value  of  m  is  the  next  problem  to  consider.  For  m  moderately 
large,  by  the  central  limit  theorem,  SIMDAT  approximately  samples  from  n  normal  distributions 
with  mean  and  covariance  matrices  corresponding  to  those  of  the  n,  m  nearest  neighbor  clouds  [52] . 
There  are  rules  for  the  consistency  of  the  non-parametric  density  estimator  that  correspond  to 
SIMDAT,  however  the  formulas  require  values  that  are  not  normally  available  [52].  However, 
the  goal  is  to  generate  new  data  points  that  resemble  those  observed,  not  construct  a  density 
estimator.  Note,  that  if  m  =  1  the  resulting  estimator  is  the  bootstrap,  and  if  m  =  n,  the  samples 
are  from  an  approximate  normal  distribution  with  mean  vector  and  covariance  matrix  as  estimated 
by  the  observed  data.  Guidelines  based  on  experience  are  to  choose  m  .02n  for  data  sizes  of 
approximately  1,000  or  larger  [52].  For  smaller  samples,  m  .05n  has  worked  well  [52]. 
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The  SIMDAT  method  can  be  applied  for  approximating  the  mean  of  the  ACV  regardless  of 
any  knowledge  of  the  moments  and/or  dependence  structure  of  x.  Experiments  using  SIMDAT  are 
performed  with  the  results  reported  later  in  this  chapter. 

5.5  Parametric  Methods 

When  generating  an  ACV,  the  inputs  to  the  analytical  model  are  output  sample  means  of 
the  simulation  model.  In  chapter  IV,  this  fact  was  used  by  the  Monte  Carlo  method  to  assume  a 
normal  distribution  for  each  of  the  elements  of  x.  For  the  purposes  of  this  section,  that  assumption 
is  still  valid.  The  difference  is  that  some,  or  all,  of  the  parameters  of  the  assumed  multivariate 
normal  distribution  are  unknown.  A  parametric  method  of  generating  new  data  points  could  be 
constructed  by  sampling  from  a  multivariate  normal  distribution  where  the  unknown  (or  aU)  the 
parameters  are  estimated  from  the  observed  data. 

Assume  that  a  random  sample  where  Xj  =  is  observed  having 

been  generated  from  the  multivariate  distribution  given  by 

Xj  ~  Normal  (/i,  E)  j  =  1, 2, . . .  ,  fe  (5.19) 

where 

M  =  (5-20) 
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and 


O'? 

<Ti2 

... 

O'lk 

<T21 

0^2 

... 

C^2k 

<^kl  **• 


(5.21) 


Further  assume  that  all  parameters  (elements  of  n  and  S)  of  the  distribution  are  unknown.  Then 
the  mean  parameters  can  be  estimated  by  fi  where 


/i  =  X  =  n-il'X  (5.22) 

where  1  is  a  n  x  1  vector  of  ones.  The  variance  and  covariance  parameters  are  estimated  by  S  given 
by 


E  =  C  =  (n  -  l)-i  [X'X  -  n“i(X'l)(l'X)]  (5.23) 

Then  N  random  vectors  can  be  generated  by  sampling  N  times  from  a  multivariate  normal  distri¬ 
bution  with  parameters  given  by  X  and  C  using  an  appropriate  random  variate  generation  scheme. 

One  simple  scheme  for  generating  multivariate  normal  random  vectors  is  provided  in  Law  and 
Kelton  [32]  attributed  to  Scheuer  and  StoUer  [46].  Assume  the  goal  is  to  generate  random  vectors 
from  a  multivariate  normal  distribution  of  dimension  k  with  mean  vector  /x  and  covariance  matrix 
E.  Since  S  is  symmetric  and  positive  definite,  it  can  be  factored  uniquely  as  S  =  CitC[^  where  the 
n  X  n  matrix  is  lower  triangular.  This  is  referred  to  as  Cholesky  factorization.  An  algorithm 
for  generating  the  required  multivariate  normal  random  vector  X  is  given  by: 

1.  Generate  the  random  vector  Z  =  (Zi,  Z2, . . .  ,  Zkf  where  each  Zi  is  distributed  as  an  i.i.d. 
Normal(0, 1)  random  variable. 
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2.  Ij6t  X  —  /x  *4"  CjfZ. 


When  all  parameters  are  known,  this  approach  is  another  way  of  performing  the  Monte 
Carlo  method  described  in  Chapter  IV.  For  the  assumption  we  made  here  (all  the  parameters  are 
unknown)  estimated  parameters  can  be  used  in  the  above  scheme. 

As  mentioned  in  the  introduction  to  this  chapter,  the  situation  is  that  all  or  some  of  the 
parameters  of  the  input  random  vector  to  the  ACV  are  unknown.  In  the  above  development,  the 
assmnption  is  that  all  parameters  are  unknown.  To  account  for  the  situation  when  some  of  the 
parameters  are  known,  the  above  scheme  can  be  modified  by  simply  replacing  the  estimated  pa- 
rameter(s)  with  the  appropriate  known  parameter (s).  For  the  unknown  parameters,  the  estimated 
parameters  are  still  used. 

By  assuming  that  is  approximately  distributed  by  a  multivariate  normal  distribution, 

the  parametric  scheme  described  above  can  be  used  to  approximate  the  mean  of  the  ACV  when 
some  (or  all)  of  the  parameters  of  Y^(j)  are  unknown.  Experiments  using  the  parametric  scheme 
are  performed  with  the  results  reported  later  in  this  chapter. 

5.4  Combined  Methods 

Another  approach  to  generating  the  random  vectors  necessary  to  approximate  the  mean  of 
the  ACV  is  to  combine  the  non-parametric  and  parametric  methods  described  above.  In  other 
words,  for  those  random  variates  that  can  be  generated  by  a  parametric  approach,  do  so.  The 
other  random  variates  could  then  be  generated  using  one  of  the  other  non-parametric  methods. 
Two  combined  methods  are  described  below. 

To  demonstrate  the  combination  methods,  consider  the  following.  Let  where  Xj  = 

{xij,X2j, ...  jXkjY  be  a  random  sample  generated  from  a  multivariate  normal  distribution  with 
mean  vector  ^  and  covariance  matrix  E  as  described  above.  Assume  that  for  some  of  the  elements 
of  X  the  mean  and  variance  is  known  and  for  the  remaining  elements,  these  expectations  are 
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unknown.  Without  loss  of  generality,  for  a  k  dimensional  vector,  let  the  first  I  <  k  components  of 
X  have  known  means  and  variances  and  the  remaining  k  —  l  components  have  unknown  means  and 
variances.  Further  assume  that  the  first  I  random  variates  are  independent  of  the  last  k  —  l  random 
variates. 

The  first  combination  method  considered  combines  the  parametric  method  of  section  5.3  with 
the  bootstrap  method  described  in  Section  5.2.1.  For  this  method,  a  further  assumption  is  made, 
namely  that  the  last  k  —  I  random  variates  are  also  independent  of  each  other.  Simply,  to  generate 
a  new  random  vector,  the  first  I  random  variates  are  generated  using  a  parametric  method  with 
known  parameters.  The  remaining  components  are  re-sampled  from  the  observed  data  using  the 
bootstrap  method. 

The  other  combination  method  joins  the  parametric  method  of  Section  5.3  with  the  SIMDAT 
method  of  Section  5.2.2.  As  with  the  other  combination  method,  the  first  I  random  variables  are 
generated  with  a  parametric  method  using  the  known  parameters.  The  other  k  —  l  variables  are 
generated  using  the  SIMDAT  method.  In  this  case,  the  last  k  -  I  variables  are  considered  to  be  a 
random  vector  of  dimension  k  —  l  for  purposes  of  the  SIMDAT  algorithm. 

Both  combination  methods  are  used  to  estimate  the  mean  of  the  ACV  later  with  results 
provided  later  in  this  chapter. 

5.5  Airfield  Operation  Example 

5,5,1  Overview.  A  discrete  event  simulation  model  of  an  AMC  airfield  is  used  to  explore 
the  effectiveness  of  approximating  the  mean  of  an  ACV  for  each  of  the  random  vector  generation 
methods  described  above.  An  MYA  nested  fork-join  analytical  model  is  also  described  for  the 
purposes  of  generating  the  ACV.  The  simulation  model  is  written  using  the  SLAM  II  simulation 
language  and  is  based  on  AMC’s  BRACE  simulation  model  [1] .  The  simulation  model  is  referred 
to  as  Pseudo^BRACE.  This  model  is  used  in  place  of  BRACE  since  it  operates  faster  and  is  easier 
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to  modify  so  is  therefore  better  suited  for  the  current  stage  of  exploratory  research.  The  section 
is  organized  in  the  following  manner.  The  simulation  model  and  the  performance  measures  of 
interests  are  described  first,  followed  by  a  description  of  the  analytical  model  and  how  it  measures 
the  same  performance  measures, 

5. 5. 2  Simulation  Model  Pseudo-BRACE  models  the  operation  of  an  US  AF  airlift  airfield. 
The  major  activities  of  such  an  airfield  are  simulated  within  the  model.  These  activities  include 
landing,  taxiing,  parking,  refueling,  scheduled  and  unscheduled  maintenance,  and  cargo  upload. 
Each  of  these  activities  are  discussed  below  with  the  aircraft  arrival  and  parking  process  discussed 
first. 

Simulated  aircraft  arrive  at  the  airfield  according  to  a  Poisson  process.  Several  different  types 
of  aircraft  are  simulated.  A  portion  of  those  aircraft  is  designated  to  carry  hazardous  cargo.  The 
airfield  has  a  finite  number  of  spots  on  its  ramp,  equal  to  P,  for  parking  the  aircraft  with  only  some 
of  the  spots  considered  safe  for  hazardous  cargo.  In  addition,  some  (not  all)  of  the  non-hazardous 
cargo  spots  are  equipped  with  fuel  hydrant  refueling  pits.  If  the  arriving  aircraft  is  designated 
for  hazardous  cargo  and  there  is  an  empty  hazardous  cargo  parking  spot,  the  aircraft  will  enter 
the  queue  of  aircraft  waiting  to  use  the  runway.  If  there  is  no  hazardous  cargo  spot  available,  the 
simulated  aircraft  will  wait  for  two  hours  for  a  hazardoios  cargo  spot  to  open  up.  If  no  spot  becomes 
available  in  that  time,  the  aircraft  will  leave  the  system;  divert  to  another  airfield.  Aircraft  not 
designated  for  hazardous  cargo  wiU  enter  the  runway  queue  when  any  parking  spot  is  available.  The 
aircraft  will  be  parked  using  the  following  preferences.  The  aircraft  will  park  at  a  spot  with  a  fuel 
hydrant  spot  first,  if  one  is  available.  If  not,  the  aircraft  will  park  in  a  non-hazardous  parking  spot 
that  doesn’t  have  a  fuel  hydrant  pit.  Finally,  the  aircraft  wall  be  parked  at  a  hazardous  parking 
spot.  Again,  if  the  aircraft  has  not  entered  the  nmway  queue  within  two  hours  of  arrival  it  wdU 
divert.  Only  one  aircraft  at  a  time  is  allowed  to  use  the  nmway.  Landing  and  taxiing  are  simulated 
using  a  fixed  amount  of  time  for  each  activity. 


5-11 


Once  the  simulated  aircraft  has  performed  a  simulated  landing  and  taxied  to  the  parking  spot, 
refueling,  cargo  upload,  and  aircraft  maintenance  are  performed  concurrently  with  one  exception. 
Servicing  the  liquid  oxygen  (LOX)  system  is  simulated  as  soon  as  the  aircraft  is  parked.  The 
simulated  servicing  lasts  a  fixed  amoimt  of  time.  No  other  simulated  activities  are  allowed  to  occur 
on  the  aircraft  xmtil  LOX  servicing  is  complete.  It  shoiild  be  pointed  out  that  other  activities  could 
begin  as  soon  as  the  aircraft  is  parked  as  long  as  they  don’t  occur  at  the  simulated  aircraft.  The 
movement  of  fuel  trucks  is  one  example. 

Both  scheduled  and  unscheduled  maintenance  is  simulated  in  the  following  way.  The  model 
assumes  that  maintenance  persoimel  are  always  available.  For  all  types  of  aircraft,  scheduled 
maintenance  is  always  performed  and  has  a  fixed  duration  that  is  the  same  for  all  aircraft  and 
begins  as  soon  as  LOX  servicing  is  complete.  Unscheduled  maintenance  duration  is  simtilated 
as  a  random  length  of  time  whose  distribution  depends  on  the  type  of  aircraft  simulated.  Each 
aircraft  type  is  assigned  a  probability  of  requiring  unscheduled  maintenance  in  one  of  8  categories: 
no  maintenance,  0-4  hours,  4-8  hours,  8-12  hours,  12-16  hours,  16-24  hours,  25-48  hours,  and 
48-72  hours.  Pseudo-BRACE  assumes  that  repair  times  are  distributed  uniformly  within  each 
time  category.  When  an  aircraft  is  parked,  its  unscheduled  maintenance  time  is  drawn  from  the 
described  random  distribution.  Unscheduled  maintenance  is  performed  concurrently  with  scheduled 
maintenance. 

Two  types  of  aircraft  refueling  are  simulated  by  Pseudo-BRACE.  They  are  hydrant  system 
refueling  and  by  refueling  truck.  If  an  aircraft  is  parked  on  parking  spot  with  a  fuel  hydrant  pit, 
the  refueling  process  is  simulated  as  a  hydrant  system  refueling.  If  there  is  more  than  one  refueling 
pit  on  the  ramp,  only  one  pit  can  be  in  operation  at  any  one  time.  In  that  case,  hydrant  refueling 
is  simulated  as  a  first-come  first-served  (FCFS)  queue  with  one  server.  A  newly  arrived  aircraft 
will  enter  the  queue  as  soon  as  LOX  servicing  is  complete.  Each  type  of  aircraft  has  a  fixed  fuel 
receive  rate  and  a  fixed  amount  of  fuel  required  upon  landing.  The  hydrant  system  also  has  a  fixed 
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rate  that  it  can  pump  fuel.  The  refueling  duration  time  is  a  fixed  time  based  on  the  amount  of  fuel 
required  and  the  lesser  of  the  two  fuel  movement  rates  and  a  fixed  amoimt  of  time  for  hooking  up 
the  system. 

Aircraft  refueling  by  truck  is  simulated  in  the  following  manner.  There  are  a  fixed  number 
of  simulated  refueling  trucks  assigned  to  the  airfield.  Each  truck  has  the  same  fixed  fuel  capacity. 
They  are  assigned  to  aircraft  parked  on  spots  without  refueling  pits  on  a  FCFS  basis.  Once  a  truck 
is  assigned  to  an  aircraft  it  is  dedicated  to  that  aircraft  until  the  aircraft  has  received  a  full  load 
of  fuel.  The  fuel  truck  can  prnnp  and  receive  fuel  at  a  fixed  rate.  Once  a  truck  becomes  available, 
refueling  an  aircraft  with  a  truck  begins  by  simulating  the  movement  of  the  truck  from  the  staging 
area  to  the  aircraft  by  a  fixed  amount  of  time.  Once  the  truck  arrives  at  the  aircraft,  and  LOX 
servicing  is  complete,  the  truck  is  hooked  up  to  the  aircraft,  simulated  by  a  fixed  amount  of  time. 
The  refueling  duration  for  a  single  truck  refueling  is  a  fixed  time  based  on  the  lesser  of  the  amount 
of  fuel  required  or  in  the  truck  with  the  pumping  rate  the  lesser  of  the  aircraft  receive  or  truck 
pump  rate.  When  the  truck  has  completed  pumping  it  travels  to  a  single  fill  stand  to  refill.  The  fill 
stand  is  simulated  as  a  FCFS  queue  with  a  single  server.  The  refill  time  is  simulated  as  before  based 
on  the  amount  of  fuel  required  to  refill  the  truck  to  capacity  and  the  lesser  of  the  fuel  movement 
rates.  If  the  aircraft  requires  more  fuel,  the  truck  returns  to  the  aircraft  and  refuels  the  aircraft 
in  the  manner  described  above.  These  activities  are  repeated  until  the  aircraft  has  received  a  full 
load  of  fuel. 

Only  the  up-load  of  cargo  is  simulated  in  Pseudo-BRACE.  The  simulated  cargo  resomces 
include  a  finite  number  of  K-loaders,  forklifts,  and  loading  docks.  An  unlimited  nmnber  of  simulated 
loaded  cargo  pallets  are  located  in  a  simulated  warehouse.  The  amoimt  of  cargo  up-loaded  on  each 
aircraft  is  a  fixed  number  of  pallets,  each  carrying  a  fixed  amount  of  cargo  (in  pounds)  that  depends 
on  the  t5rpe  of  aircraft  simulated.  Upon  parking,  an  aircraft  enters  a  simulated  FCFS  queue  for  an 
available  loading  dock.  Once  a  loading  dock  becomes  available,  any  available  forklifts  move  loaded 
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cargo  pallets  from  the  warehouse  to  the  dedicated  loading  dock.  The  movement  time  is  fixed.  A 
single  loading  dock  is  large  enough  to  hold  all  the  pallets  for  one  aircraft.  Once  the  pallets  are 
on  the  dock,  the  aircraft  enters  a  simulated  FCFS  queue  for  K-loaders.  K-loaders  can  hold  up  to 
five  pallets  each.  If  more  than  one  K-loader  is  required  to  move  all  the  cargo  for  one  aircraft  they 
will  be  assigned  to  the  aircraft  as  soon  as  they  become  available.  The  time  to  load  each  pallet  on 
a  K-loader  and  the  time  it  takes  for  a  K-loader  to  get  to  an  aircraft  are  all  simulated  by  a  fixed 
amoimt  of  time.  Once  a  K-loader  arrives  at  an  aircraft  fixed  set-up  and  separate  pallet  up-load 
times  are  simulated,  given  that  LOX  servicing  is  complete.  The  first  K-loader  to  arrive  at  an 
aircraft  incurs  an  additional  fixed  “manifest  processing”  simulated  time.  This  operation  continues 
untU  the  aircraft  has  received  its  full  cargo  load. 

Before  departing  an  airfield,  each  simulated  aircraft  must  spend  a  minimum  amount  of  time 
at  the  airfield,  even  if  all  servicing  and  cargo  up-load  activities  are  complete.  The  minimum  time  is 
called  a  standard  ground  time.  This  time  represents  the  standard  time  used  by  operations  planners 
for  scheduling  aircrews.  Once  all  servicing  has  been  completed  and  the  standard  ground  time  has 
been  exceeded,  aircraft  taxi  and  enter  the  nmway  queue.  Once  the  runway  becomes  available,  the 
aircraft  simulates  a  take-off  and  departs  the  system. 

Many  different  performance  measures  for  this  simulation  are  of  interest  to  decision-makers. 
For  the  purposes  of  this  research  two  statistics  will  be  gathered,  the  mean  steady  state  turn  time, 
r,  and  the  mean  steady  state  sojourn  time,  p.  Turn  time  is  defined  as  the  amoimt  of  time  it 
takes  an  aircraft,  once  it  is  parked,  for  it  to  be  refueled,  have  all  maintenance  completed,  and 
all  cargo  up-loaded.  Sojourn  time  is  defined  as  the  maximum  of  turn  time  and  standard  groimd 
time.  To  estimate  the  performance  measures  n  independent  replications  are  performed.  These 
steady-state  performance  measures  are  estimated  by  tnmcating  a  constant  number  of  events  from 
the  beginning  of  each  simulation  replication  to  eliminate  initial  transient  behavior  bias.  The  reader 
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should  interpret  all  definitions  of  the  following  statistics  to  implicitly  include  these  truncations.  Let 
N  represent  the  number  of  simulated  aircraft  that  enter  the  airfield  and  receive  servicing. 

To  estimate  r  let  %  be  the  amoimt  of  simulated  turn  time  for  the  i*'"  aircraft  arriving  to  the 
airfield  during  the  replication.  Then  the  mean  turn  time  for  replication  j  is 

=  j  =  (5.24) 

i=l 

SO  that  T  is  estimated  by 

f  =  f=-Vr,  (5.25) 

n  " 

3=1 

In  a  similar  manner,  to  estimate  p,  let  rij  represent  the  sojourn  time  of  the  aircraft  to  arrive  on 

the  airfield  during  the  replication.  The  mean  sojourn  time  for  replication  j  is  foimd  by 

1  ^ 

Rj  =  J  =  1, 2, . . .  , n  (5.26) 

i=:l 

The  estimate  of  p  is  then 

P  =  R  =  lj2R3  (5-27) 

n 

3-1 

5.5.3  Analytical  Model  To  construct  an  anal37tical  model,  begin  by  considering  a  single 
class  capacitated  open  queuing  network  depicted  in  Figure  5.1,  with  capacity  P.  This  model  is  based 
on  the  Pseudo-BRACE  simidation  model.  In  the  analytical  model,  the  aircraft  are  represented  as 
customers  that  require  service  at  9  separate  stations  (queues)  in  the  model.  All  stations  use  the 
FCFS  service  discipline  and  the  mean  service  times  at  each  station  are  derived  from  the  mean  times 
in  Pseudo-BRACE. 
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3.Mx 


Poisson  arrival  process  {  rate  *  X) 


Figure  5.1  Open  queueuing  network. 


The  stations  model  activities  that  occur  within  the  Pseudo-BRACE  model.  The  mean  service 
time  at  each  station  is  determined  by  the  mean  times  required  by  PseudoBRACE  to  complete 
the  modeled  activity.  The  first  station  is  a  single  server  queue  with  an  exponential  service  time 
representing  the  landing  of  the  aircraft  at  the  airfield.  Taxiing  to  a  parking  spot  is  represented  by 
the  second  station  which  has  an  infinite  server  and  a  deterministic  service  time.  The  third  station 
represents  scheduled  and  unschediiled  maintenance  with  an  infinite  server  and  exponential  seiwice 
time.  Cargo  upload  occurs  at  station  4.  The  time  required  to  up-load  cargo  is  represented  by 
an  exponential  random  variable  and  the  number  of  servers  is  equal  to  the  number  of  K-loaders  in 
Pseudo-BRACE  divided  by  the  number  of  K-loaders  necessary  to  up-load  a  single  aircraft.  Non- 
concurrent  maintenance  is  modeled  by  station  5  with  an  exponential  service  time  and  an  infinite 
number  of  servers.  Hydrant  refueling  is  at  the  sixth  station  which  consists  of  a  single  server  with  an 
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exponential  service  time.  Station  7  models  aircraft  refueling  by  truck  with  the  number  of  servers,  t, 
equal  to  the  number  of  trucks  in  the  Pseudo-BRACE  model.  Service  time  is  an  exponential  random 
variable.  The  eighth  station  models  a  departing  aircraft  taxiing  to  the  runway  and  has  an  infinite 
server  with  a  deterministic  service  time.  The  ninth  and  last  station  models  aircraft  take-oflF  with  a 
single  server  queue  with  an  exponential  service  time. 

Due  to  the  concurrent  servicing  performed  in  the  simulation  model,  the  analytical  model 
contains  fork-join  constructs  depicted  as  diamonds  in  Figure  5.1.  As  described  in  Chapter  11, 
multiple  activities  are  performed  concurrently  within  the  fork-join  constructs.  The  main  fork-join 
node  is  designated  by  A  with  paths  Al,  A2,  and  A3.  A  second  fork-join  path  is  designated  as  B  with 
paths  B1  and  B2.  Note  that  LOX  servicing  is  included  in  path  Bl.  Although  in  Pseudo-BRACE, 
LOX  servicing  must  be  completed  before  any  other  servicing  can  begin,  the  movement  of  the  fuel 
trucks  and  pallets  to  the  aircraft  can  begin  immediately  since  it  is  not  possible  for  them  to  arrive 
at  the  aircraft  before  LOX  servicing  is  complete.  Also,  since  aircraft  maintenance  is  performed  by 
an  infinite  server,  the  time  for  LOX  servicing  is  added  into  the  total  time  for  maintenance. 

In  order  to  apply  the  MYA  algorithm  and  fork-join  heuristic  a  modification  is  made  to  the 
model.  The  modified  network  is  shown  in  Figure  5.2.  The  open  capacitated  system  is  transformed 
into  an  equivalent  closed  network.  The  transformation  is  accomplished  by  adding  a  new  single 
server  queue,  station  0,  with  an  exponential  service  time  to  the  system  and  setting  the  number  of 
customers,  P,  in  the  system  equal  to  the  capacity  of  the  original  open  network.  The  new  “arrival” 
station  represents  that  portion  of  the  airlift  system  that  operates  outside  of  the  airfield.  When  all 
P  customers  are  in  the  airfield  portion  of  the  network,  no  new  arrivals  can  be  generated.  This  is 
equivalent  to  a  fully  populated  open  capacitated  network.  Further,  if  the  airfield  is  not  at  capacity, 
at  least  one  customer  is  in  the  arrival  queue  so  that  arrivals  are  generated  by  a  Poisson  process 
with  rate  of  1  over  the  mean  of  the  exponential  service  time.  This  model  does  not  account  for 
the  2  hour  time  period  that  an  aircraft  will  spend  waiting  for  a  parking  spot  in  Pseudo-BRACE. 
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The  assumption  of  the  aiial3rtical  model  is  that  any  arriving  aircraft  that  finds  the  airfield  fiiU  will 
divert.  Also,  hazardous  cargo  and  the  separate  hazardous  cargo  parking  spots  are  not  modeled. 

3.Mx 


1.  Land  0.  Arrival  9.  Take  off 

E/1  E/1  E/1 


Figure  5.2  Closed  queueuing  network. 

Recall  from  Chapter  II  that  to  apply  the  fork-join  heuristic,  the  MVA  algorithm  is  modified 
by  applying  the  fork-join  approximations  and  conditioning  on  the  sets  of  fork-join  paths  taken  [17]. 
Prom  Figure  5.2  it  is  seen  that  the  path  transit  times  for  fork-join  node  B  when  N  customers  are 
in  the  network  are  given  by 


E[Tbi{N)] 

=  R5{N)  +  Ik{N) 

(5.28) 

E[TB2iN)] 

=  R7{N) 

(5.29) 
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where  i  =  0, 1, 2, . . .  ,  9  are  the  station  response  times  found  by  the  MVA  aJgorithm  when 

N  customers  are  in  the  network.  By  approximation  2,  Tj{N)  is  approximated  by  an  exponential 
random  variable  with  rate  parameter  0i{N)  =  l/£J[3j(iV)].  Using  approximation  2,  the  mean 
holding  time  for  fork-join  node  B  is  given  by  [1"^ 

+  (5-30) 

Tn  the  same  way,  path  transit  times  for  fork-join  node  A  are  approximated  by 


E[TAim 

=  RziN) 

(5.31) 

E[Ta2{N)] 

=  MN) 

(5.32) 

E[Ta3{N)] 

=  £;[rB(iv)] 

(5.33) 

Mean  holding  time  for  fork-join  node  A  is  given  by 


[  ^(  )]  eAsiN) 

1  1  1 
0AliN)  +  eA2{N)  eAl{N)+0A3{N)  eA2{N)  +  9A3{N) 

eAi{N)  +  eA2{N)  +  eA3{N)} 


(5.34) 


Applying  approximation  1,  the  cycle  time  for  the  network  can  then  be  computed  by  the  MVA 
algorithm  for  N  as  [17] 


2 


9 


CTo{N)  «  ^Jli(iV)  +  rA(iV)  + 

1=1  i=8 


(5.35) 


An  additional  modification  is  made  to  the  model,  depicted  in  Figure  5.3,  by  adding  station 
H.  Station  H  represents  the  standard  ground  time,  SGT,  each  aircraft  must  spend  on  the  ground. 
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This  station  has  an  infinite  number  of  servers  and  a  deterministic  service  time.  It  is  placed  on  path 
AH  and  the  solution  to  the  fork-join  heuristic  must  be  changed  to  accoimt  for  its  behavior.  If  an 
aircraft  traverses  all  other  paths  in  fork-join  node  A  in  less  time  than  SGT,  the  time  spent  in  fork- 
join  node  A  is  equal  to  SGT.  On  the  other  hand,  if  the  maximum  time  spent  in  one,  or  all,  of  the 
other  paths  is  a  >  SGT,  the  time  spent  in  fork-join  node  A  is  a.  Using  the  memoryless  property  of 
the  exponential  random  variable,  and  conditioning  on  all  possible  outcomes,  the  fork-join  heuristic 
is  easily  modified  to  account  for  station  AH, 

Consider  the  3  exponential  random  variables  Tax{N),  Ta2{^),  and  r^3(^)  that  represent  the 
time  an  aircraft  clone  spends  in  paths  ^41,^2,  and  A^  respectively.  Let  the  probability 

that  TAi{N)  >  SGT,  i  =  1,2,3,  which  is  found  by 

pSGT 

=  1  ~  /  eAi{N)exp {-eAi{N)t} dt  =  exp {-~eAi{N)SGT}  (5.36) 

Jo 

It  is  possible  that  an  aircraft  clone  spends  more  than  SGT  in  none,  all,  one,  or  several  of  the  three 
A  paths.  Let  Qa  be  the  union  of  all  subsets  of  possible  paths  greater  than  SGT,  where  there 
are  2^  =  8  different  subsets.  The  subsets  are  represented  by  5j,  j  =  1, 2, . . .  ,  8.  Let  tTj  be  the 
probability  that  subset  Sj  occurs.  Since  each  path  is  independent  of  all  the  others,  tTj  is  easily 
calculated  by 


(5.37) 

i^Sj  i^Sj 

Given  that  a  particular  Sj  occurs,  the  mean  holding  time  is  adjusted  in  the  following  man¬ 
ner.  Recall  that  computation  of  mean  holding  time  is  mathematically  equivalent  to  determining  the 
mean  time  to  failure  for  a  parallel  system  of  independent  components  with  exponentially  distributed 
failure  times  [17].  By  the  memoryless  property  of  the  exponential  distribution,  the  expected  value 
of  an  exponential  random  variable  given  that  its  value  is  greater  than  b  is  simply  b  plus  the  imcon- 
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1.  Land  0.  Arrival  9.  Take  off 

E/1  E/1  E/I 

Figure  5.3  Closed  queueing  network  with  standard  ground  time  station  H 


ditional  expected  value  [43].  Therefore,  assuming  that  some  of  the  components  have  not  failed  by  a 
given  time  5,  the  expected  time  imtil  failure  is  b  plus  unconditional  mean  time  imtil  failure.  Hence, 
for  this  network,  mean  holding  time  for  path  Sj  is  designated  by  E[in.ecxi^Sj  {TAi{N)HOLD}]  and 
is  given  by 

E[m^{TAi{N)HOLD}]  =  5GT  +  £[max{T^i(iV)}] 


(5.38) 


where  Efmaxies^  {TAi(iV)}]  is  calculated  as  before.  Then  the  mean  holding  time  for  fork-join  path 
is  foimd  by  conditioning  on  the  different  possible  paths  by 

E[TAiN)HOLD]  =  [7ri,7r2,...'?r8] 

-|T 

. ..  ,E[m^{TAiiN)HOLD}]  (5.39) 

Then  to  complete  the  MVA  algorithm,  the  cycle  time  for  the  network  is  computed  as 

2  9 

CTo{N)  «  '^Ri{N)  +  Ta{N)hold  +  (5.40) 

i=l  i=8 

The  performance  measures  for  the  analytical  model  that  correspond  to  the  simulation  per¬ 
formance  measures  turn  time  and  sojourn  time  are  Ta{P)  and  Ta{P)hold  respectively,  where 
P  is  the  total  number  of  parking  spots  in  Pseudo-BRACE.  To  generate  the  necessary  ACV’s  let 
Yacj)  =  where  is  the  sample  mean  aircraft  interarrival  time, 

y^(2i)  ^  sample  mean  maintenance  time,  Ya^sj)  ^  observed  proportion  of  aircraft  that  are 
refueled  by  hydrants,  and  5^(4^)  is  the  sample  mean  cargo  up-load  time  for  rephcation  j.  No  other 
variables  need  be  passed  to  the  analytical  model  since  the  other  activities  simulated  in  Pseudo- 
BRACE  last  a  fixed  amount  of  time.  Due  to  the  nature  of  Pseudo-BRACE,  the  expected  values 
and  variances  of  Y^^ij)  Y^(^2j)  known  since  they  are  functions  of  input  stochastic  processes. 

On  the  other  hand,  those  of  5^(3^)  scad  Y^^^j)  known  since  they  are  strictly  outputs  of 

the  simulation  model.  Further,  it  is  known  that  the  interarrival  times  and  maintenance  times  are 
independent  of  each  other.  AU  other  covariances  are  unknown.  These  observed  means  are  then 
used  to  calculate  Ta{P)  (which  is  not  used  later  in  the  MVA  algorithm  due  to  the  nature  of  the 
final  network)  and  Ta{P)hold- 
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5.6  Performance  Comparisons 


5.6.1  Experimental  Procedures.  Experiments  are  conducted  to  compare  the  different 
methods  of  estimating  the  ACV  mean.  One  network  design  point  is  selected.  The  selection  of 
parameters  is  discussed  in  a  later  section.  Two  types  of  studies  are  performed  in  order  to  compare 
the  different  random  vector  generation  schemes.  For  eadi  experiment  in  both  studies,  1,000  ran¬ 
dom  vectors  are  generated  to  find  by  each  generation  scheme.  The  first  study  consists  of  50 
experiments  that  compare  the  different  approximated  values  of  with  the  actual  value  of  /x^.  For 
each  experiment,  20  consecutive  replications  are  chosen  at  random  from  a  set  of  10,000  replications. 
The  20  replications  are  used  to  estimate  parameters  or  for  re-sampling  for  each  of  the  different  ran¬ 
dom  vector  generation  schemes.  Comparisons  are  made  using  relative  absolute  error  percentage, 
coverage,  and  MSE.  The  second  study  compares  the  coverage  and  MSE  observed  for  controlled 
estimates  of  the  performance  measures  using  25  experiments  with  the  number  of  simulation  model 
replications  equal  to  20.  In  this  case,  the  observed  data  from  the  appropriate  set  of  simulation 
replications  is  used  to  generate  the  necessary  random  vectors. 

Comparisons  for  the  first  study  are  made  in  the  following  manner.  Let  actual 

expected  value  of  the  ACV.  For  the  1,000  random  vectors  generated  for  the  experiment,  h  = 
1,2,...  ,  50,  an  estimate  of  is  computed.  Call  the  estimate  Pz(h)Q)y  where  Z  =  1, 2, . . .  ,6,  denote 
the  6  different  random  vector  generation  schemes  used.  The  methods  are  described  later  in  the 
next  section.  Then  relative  absolute  error,  Eh{l)  for  the  experiment  using  the  scheme  is 
calculated  by 


Eh{1)  = 


jj’Zjh)  (0  ~  Mz 


(5.41) 
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The  relative  absolute  error  percentage  for  method  I  is  then  given  by 


-  50 

Eil)%  =  (100%)^  ^Eh(l)  l  =  l,2,...,Q 


(5.42) 


Another  means  of  assessing  the  accuracy  of  the  dijHFerent  schemes  for  approximating  the  ACV 
mean  is  to  compare  realized  coverage  for  each  method.  Begin  by  letting  d^{l)  denote  the  6  different 
estimates  of  the  variance  of  For  the  experiment,  the  confidence  interval  estimate  is 

given  by 


(5.43) 


where  Hk{l)  is  the  estimated  half-width  given  by 


Hh{l)  =  ^l-a/2,999 


(5.44) 


with  a  =  0.10.  The  estimated  confidence  interval  coverage  probability  is  found  by  first  letting 


r 


1 

0 


if  €  Hh{l) 
otherwise 


(5.45) 


for  Z  =  1, 2, . . .  ,  6  and  h—  1,2,...  ,50.  The  estimate  of  the  confidence  interval  coverage  is  given 
by  the  calcidated  coverage  fraction  for  Ah{l)y  computed  as 


^  50 
h=l 


(5.46) 


Realized  coverage  is  not  always  completely  indicative  of  the  accuracy  of  an  approximation. 
For  example  a  point  estimate  may  be  very  close  to  but  if  the  associated  confidence  interval 
is  small,  coverage  may  not  be  realized.  Another  measure  of  accuracy  that  considers  this  is  the 
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estimated  value  of  the  MSE  of  point  estimator.  The  estimated  MSE  is  computed  by 


1  \  2 

MSE{1)  =  il)-f^z)  /  =  1, 2, . . .  , 6  (5.47) 

The  actual  expected  values  of  the  ACV  for  both  turn  time  and  sojourn  time  are  estimated 
by  performing  10,000  replications  of  Pseudo-BRACE  and  finding  the  ACV  for  each  of  those  repli¬ 
cations.  The  resulting  sample  means  are  used  to  estimate  for  both  performance  measures. 

The  second  study  compares  estimated  coverage  and  MSE  for  the  different  schemes,  but  in 
this  case  the  comparison  is  made  using  the  ACV  controlled  performance  measures.  For  each  of 
the  25  experiments,  the  6  different  estimates  of  generated  by  the  different  methods  are  used 
to  produce  the  controlled  estimates.  In  addition,  to  achieve  the  10%  confidence  interval  size  for 
fi^  compared  to  the  controlled  response,  4,000  random  vectors  are  generated  for  each  experiment. 
The  coverage  and  MSE  comparisons  are  then  made  using  the  same  procedures  outlined  above  with 
d  =  25  and  using  the  expected  value  of  the  performance  measure,  in  place  of  Also, 
the  controlled  estimate  of  for  the  experiment  using  the  different  random  vector  generation 
scheme,  is  used  instead  of  Further  the  confidence  interval  half  width  is  estimated  using 

Equation  (2.21).  The  expected  values  of  the  performance  measures  are  estimated  using  the  same 
10,000  replications  described  above. 

5.6.2  Random  Vector  Generation  Schemes.  The  following  random  vector  generation 
schemes  are  used.  Each  is  based  on  the  methods  described  in  Sections  5.2,  5.3,  and  5.4.  Recall 
that  the  input  to  the  analytical  model,  that  must  be  approximated  by  these  schemes,  is  for 

replication  j  where  ^  sample  mean  aircraft  interarrival  time,  ^  sample  mean 

maintenance  time,  ^  observed  proportion  of  aircraft  that  are  refueled  by  hydrants,  and 

^A{4:j)  sample  mean  cargo  up-load  time.  The  first  scheme  examined  is  the  non-parametric 

SIMDAT  method.  The  parametric  method  described  in  Section  5.3  is  used  to  create  3  different 
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schemes  for  generating  random  vectors  that  depend  on  our  knowledge  of  the  random  variate  param¬ 
eters.  The  first  is  called  the  NORM-EST  method  since  we  assume  all  parameters  of  the  multivariate 
normal  distribution  are  unknown  and  are  thus  estimated  by  the  observed  data.  On  the  other  hand, 
NORM-MU  assumes  that  we  know  means  of  other  means  and  all  values 

of  the  covariance  matrix  are  then  estimated.  NORM- ALL  uses  all  known  parameters — ^the  means 
and  variances  of  aircraft  interarrival  and  maintenance  times  and  the  zero  covariance  values  between 
these  two  variables — ^when  generating  random  vectors.  All  other  parameters  are  estimated. 

The  combination  methods  used  are  described  in  Section  5.4.  They  are  referred  to  here  as 
the  NORM-BOOT  and  the  NORM-SIMDAT.  In  both  combined  methods,  Yiiij) 
generated  via  the  parametric  method  using  the  known  parameters,  and  ^^(3^)  and  are 

generated  with  a  non-parametric  method.  Also,  both  combined  methods  assume  that  and 

are  independent  of  and  ^(4j)  even  though  this  is  not  true.  For  the  NORM-BOOT,  it 
is  further  assumed  that  T^(3j)  and  are  independent  of  each  other.  Based  on  that  assumption, 

the  bootstrap  method  is  used  to  re-sample  the  observed  data  and  generate  new  random  vectors. 
The  NORM-SIMDAT  uses  the  SIMDAT  method  to  generate  new  pseudo-data  points  for  ^^(3^)  and 
•  In  tbat  case,  no  assumption  about  their  dependence  structure  is  necessary. 

5,6.3  Network  Settings,  One  experimental  design  setting  is  used  to  conduct  the  exper¬ 
iments  described  above.  Table  5.1  describes  the  number  and  types  of  resources  used  in  Pseudo- 
BRACE.  Other  network  settings  include  the  length  of  the  standard  ground  time,  2.25  hours,  and 
the  length  of  time  an  aircraft  will  wait  before  diverting,  2  hours.  Recall  that  an  aircraft  without 
hazardous  cargo  will  be  parked  at  a  spot  with  a  fuel  hydrant  pit,  if  one  is  available.  If  not,  it  will  be 
parked  first  at  a  non-hazardous  parking  spot  and  then  a  hazardous  parking  spot.  The  probability 
that  an  arriving  aircraft  will  up-load  hazardous  cargo  is  0.10. 

Aircraft  arrive  to  the  airfield  via  a  simulated  Poisson  process  with  mean  arrival  rate  of  1.25 
aircraft  every  horn:.  One  thousand  aircraft  arrive  at  the  airfield  during  each  simulation  replication. 
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Table  5.1  PseudoBRACE  resources. 


Resource 

Number 

Total  Parking  spots 

12 

Hydrant  parking  spots 

4 

Hazardous  parking  spots 

4 

Fuel  Trucks 

8 

Fuel  fill  stand 

1 

K-Loaders 

8 

Forklifts 

19 

Loading  docks 

10 

To  eliminate  the  ejffects  of  initial  transient  bias,  statistics  are  gathered  on  the  last  900  arrivals.  The 
diflferent  aircraft  types  and  their  parameters  are  listed  below  in  Table  5.2.  Included  in  Table  5.2 
are  the  probabilities  that  an  aircraft  arrival  will  be  of  a  particular  type.  Based  on  the  fuel  load 
requirements  and  fuel  truck  capacities,  fuel  trucks  will  have  to  make  2  trips  to  an  aircraft  to  refuel 
it.  The  cargo  up-load  requirement  and  average  pallet  cargo  weight  results  in  a  total  of  12  pallets 
for  every  aircraft  up-load.  This  translates  to  3  K-loaders  (K-loader  capacity  equals  5  pallets)  for 
every  aircraft.  The  aircraft  unscheduled  maintenance  category  probabilities  are  recorded  in  Table 
5.3.  Within  each  category,  the  imscheduled  maintenance  time  is  uniformly  distributed.  Scheduled 
maintenance  (performed  concurrently  with  unscheduled  maintenance)  is  fixed  at  30  minutes  and 
LOX  servicing  is  fixed  at  9  minutes. 


Table  5.2  Pseudo-BRACE  aircraft  parameters. 


Type 

Probability 

Fuel  load 
(gals) 

Cargo  load 
(tons) 

Fuel  receive 
rate  (gpm) 

C-130 

0.30 

10000 

26.0 

450.0 

C-17 

0.20 

10000 

26.0 

450.0 

C-5 

0.20 

10000 

26.0 

450.0 

C-141 

0.15 

10000 

26.0 

450.0 

B-747 

0.05 

10000 

26.0 

450.0 

DC-8 

0.10 

10000 

26.0 

450.0 

The  known  mean  and  variance  for  maintenance  times  are  given  by  =  1.6921  hours, 

=  0.03327  hours^.  The  maintenance  time  includes  LOX  servicing  and  scheduled  and 
unscheduled  maintenance.  The  mean  and  variance  for  the  sample  mean  interarrival  times  are 
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Table  5.3  PseudoBRACE  unschedtiled  maintenance  probabilities. 


Time  category  (hours) 


Type 

0-4 

4-8 

8-12 

12-16 

16-24 

24-48 

48-72 

None 

C-130 

0.000 

0.000 

0.000 

0.000 

0.000 

0.000 

0.000 

1.000 

C-17 

0.000 

0.010 

0.000 

0.000 

0.010 

0.000 

0.010 

0.930 

C-5 

0.043 

0.057 

0.029 

0.036 

0.021 

0.021 

0.007 

0.786 

C-141 

0.033 

0.030 

0.027 

0.033 

0.017 

0.020 

0.003 

0.837 

B-747 

0.000 

0.000 

0.000 

0.000 

0.000 

0.000 

0.000 

1.000 

DC-8 

0.000 

0.000 

0.000 

0.000 

0.000 

0.000 

0.000 

1.000 

£;[Y^(2i)]  =  0*8  hours  and  Var[Y^^^j>^]  =  0.0007111  hours^.  Also,  it  is  known  that  and 
are  independent  of  each  other.  The  other  elements  of  the  covariance  matrix  axe  unknown. 


5.64  Results.  The  first  study  resiilts  indicate  that  the  diflFerent  schemes  have  varying 
levels  of  success  in  approximating  The  results  axe  reported  in  tables  5.4  and  5.5.  The  relative 
absolute  percentage  error  and  MSE  indicate  approximations  that  axe  very  close  to  the  actual  ACV 
mean  for  all  methods  explored.  However,  the  coverage  estimates  axe  well  below  the  nominal  value 
of  0.90.  This  is  an  indication  that  the  confidence  intervals  are  very  small  when  compared  to  the  size 
of  fiz'  hi  fact,  the  reported  average  confidence  interval  width  for  aU  methods  for  both  performance 
measures  axe  all  approximately  0.01  while  the  values  oi  about  2.9  and  3.3  for  turn  time  and 

sojourn  time  respectively.  The  most  striking  outcome  is  the  two  groups  formed  based  on  results. 
One  group  consists  of  the  methods  that  estimate  all  parameters  and  the  other  group  is  formed  of 
the  methods  that  use  some  known  parameters.  The  results  within  each  group  are  very  simUax.  The 
group  where  all  parameters  axe  estimated,  which  are  SIMDAT  and  NORM-EST,  is  also  the  group 
that  produces  the  largest  errors  and  smallest  coverage  estimates.  The  other  group  that  uses  known 
parameters  performs  markedly  better. 

Results  from  the  second  study  appear  in  Tables  5.6  and  5.7.  The  Tables  indicate  that  the 
grouping  of  the  methods  based  on  results  is  the  same  as  that  of  the  first  study.  The  methods  that 
use  known  parameters  perform  demonstrably  better  with  the  two  combined  methods  providing 
marginally  better  results  than  the  completely  parametric  methods.  Note  that  the  coverage  estimates 
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Table  5.4  ACV  mean  approximatioHj  comparisons  (Turn  time). 


Turn  time 

Method 

Mean 

Relative  absolute 
error  (%) 

Coverage  (%) 

MSE 
(  xlO-*^) 

SIMDAT 

2.8671 

2.79 

0.14 

11.6518 

NORM-EST 

2.8721 

2.69 

12.1099 

0.79 

1.0418 

■afanici 

0.78 

HKEUHI 

NORM-BOOT 

2.8709 

0.67 

NORM-SIMDAT 

2.8694 

0.72 

0.54 

0.7717 

Actual  Mean 

2.8712 

Table  5.5  ACV  mean  approximation,  comparisons  (Sojourn  Time). 


Sojourn  time 

Method 

Mean 

Relative  absolute 
error  (%) 

Coverage  (%) 

MSE 
(  xlO-'^) 

SIMDAT 

2.22 

0.12 

7.4457 

NORM-EST 

K|BI 

2.15 

7.7651 

NORM-MU 

3.2499 

0.64 

0.54 

NORM-ALL 

0.61 

0.52 

NORM-BOOT 

3.2479 

HIHBSSliHi 

0.56 

NORM-SIMDAT 

3.2467 

0.52 

Actual  Mean 

3.2484 

are  slightly  lower  than  nominal  for  some  of  the  better  performing  methods.  This  is  attributable  to 
the  extremely  small  realized  confidence  interval  widths  for  the  controlled  responses  of  approximately 
0.044  for  turn  time  and  0.052  for  sojourn  time.  Further,  the  realized  coverage  for  the  uncontrolled 
responses  is  only  84%  for  both  responses.  Hence,  in  some  cases,  the  ACV  method  actually  increases 
the  realized  coverage.  Therefore,  the  extremely  small  MSE  is  a  better  indicator  of  the  small 
observable  bias  in  the  better  performing  methods.  To  achieve  the  same  results  for  SEMDAT  and 
NORM-EST  additional  random  vectors  need  to  be  generated.  Also,  it  should  be  noted  that  the 
ACV  performs  well  in  reducing  variance  and  the  resulting  confidence  interval  widths.  For  turn 
time,  the  confidence  interval  width  reduction  is  approximately  70%  and  61%  for  sojourn  time. 
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Table  5.6  Controlled  response  comparisons  (Turn  time). 


Turn  time 

Method 

Mean 

Coverage  (%) 

MSE 
(  xlO-^) 

SIMDAT 

2.6825 

0.40 

35.5351 

NORM-EST 

2.6857 

0.40 

32.8867 

NORM-MU 

2.6613 

0.70 

3.6922 

NORM-ALL 

2.6624 

0.85 

3.2714 

NORM-BOOT 

2.6582 

0.75 

3.3086 

NORM-SIMDAT 

2.6597 

0.80 

2.9380 

Actual  Mean 

2.6556 

Table  5.7  Controlled  response  comparisons  (Sojourn  time). 


Sojourn  time 

Method 

Mean 

Coverage  (%) 

MSE 
(  xlO-^) 

SIMDAT 

3.2798 

0.50 

30.0825 

NORM-EST 

3.2838 

0.50 

28.4189 

NORM-MU 

3.2620 

0.95 

3.1823 

NORM-ALL 

3.2632 

0.95 

3.9197 

NORM-BOOT 

3.2589 

0.90 

2.4078 

NORM-SIMDAT 

3.2601 

0.95 

2.0298 

Actual  Mean 

3.2563 

5. 7  Conclusion 

This  research  into  different  non-parametric  and  parametric  methods  for  approximating  the 
mean  of  the  ACV  confirms  that  the  methods  examined  have  merit  and  can  be  used  to  reduce  the 
observed  bias  in  ACV  controlled  responses.  Despite  the  lack  of  knowledge  of  certain  parameters  of 
this  research  has  demonstrated  several  methods  that  generate  random  vectors  that  mimic 
for  the  Psuedo>BRACE  simulation  model.  These  methods  allow  us  to  continue  researdi  into 
other  areas  that  rely  upon  the  ACV  method — ^namely  surrogate  search — using  the  Psuedo-BRACE 
simulation  model. 
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VL  Surrogate  Search  Methods 


6.1  Overview 

We  have  shown  that  the  ACV  Monte  Carlo  method  efficiently  reduces  the  variance  and  confi¬ 
dence  interval  width  of  simulation  performance  measures  for  two  different  simulation  models.  Hence 
the  potential  of  reducing  simulation  study  times  using  an  external  analytical  model  is  demonstrated 
with  the  application  of  the  methods  discussed  thus  far.  The  next  step  is  to  show  how  the  ACV 
method  of  variance  reduction  can  be  used  as  a  starting  point  for  a  new  method  of  reducing  simula¬ 
tion  study  times.  The  new  method  consists  of  computing  performance  measures  from  an  analytical 
model  instead  of  performing  simulation  replications  in  order  to  estimate  system  performance  mea¬ 
sures.  Since  successful  ACV  application  requires  an  analytical  model  that  can  be  computed  much 
more  rapidly  than  the  simulation  model,  significant  time  savings  can  be  achieved. 

We  demonstrate  a  method  for  validating  and  using  an  analytical  model  as  a  surrogate  of  a 
simulation  model  to  perform  surrogate  searches  of  a  simulation  experimental  design  space.  We  use 
response  surface  methodology  (RSM)  as  the  context  for  the  surrogate  search  method.  It  is  important 
to  point  out,  that  the  intention  is  not  to  ehminate  the  simulation  model.  Rather,  the  goal  is  to 
augment,  or  enhance,  the  performance  of  the  simulation  model  with  an  analytical  model  that  has 
been  validated  for  the  specific  purpose  of  performing  the  surrogate  search.  In  general,  a  valid  model 
is  one  that  is  an  accurate  representation  of  the  system  under  study  [32].  The  specific  definition  of 
analytical  model  validation  for  the  purposes  of  performing  a  surrogate  search  is  described  below  in 
section  6.3.  The  validation  process  is  performed  in  conjunction  with  the  application  of  the  ACV 
method  during  as  a  RSM  simulation  study  design  of  experiment  is  realized. 

During  a  surrogate  search,  the  analytical  model  performs  a  search  of  a  design  space  where  the 
expected  responses  of  the  simulation  model  are  unknown,  in  order  to  identify  interesting  (locally 
optimal  and/or  nearly  optimal)  design  points  for  further  investigation.  Using  the  surrogate  search 
results  as  a  guide,  the  simulation  model  is  replicated  at  these  interesting  points  in  order  to  validate 
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the  surrogate  search  results.  If  the  analytical  model  is  an  accurate  predictor  of  the  simulation 
model,  only  those  validation  replications  are  necessary  in  order  to  establish  a  new  experimental 
design  for  the  simulation  study.  Significant  time  savings  are  then  realized  as  it  is  not  necessary  to 
perform  costly  simiilation  replications  at  uninteresting  points  along  the  search  path. 

The  chapter  is  organized  in  the  following  manner.  The  first  section  briefly  describes  a  common 
simulation  model  verification  and  validation  process.  This  process  is  modified  in  the  following 
section  as  a  new  validation  process  in  order  to  justify  (validate)  an  analytical  model  for  the  surrogate 
search  method.  The  final  section  of  the  chapter  develops  the  surrogate  search  method. 

6.2  Simulation  Model  Verification  and  Validation 

Simulation  models  are  used  throughout  industry  and  the  military  to  provide  decision-makers 
with  the  information  necessary  to  make  informed  decisions  concerning  complex  systems  and  prob¬ 
lems.  The  correctness  of  these  decisions  depends  on  the  accuracy  of  the  simulation  models.  Ver¬ 
ification  and  validation  methods  are  employed  by  simulation  developers  and  analysts  in  order  to 
determine  the  accuracy  of  simulation  models  [45].  Several  authors  have  proposed  definitions  for 
verification  and  validation.  Two  of  the  more  commonly  accepted  definitions  for  model  validation 
follow  [32,45].  Model  validation  is  often  defined  as  “substantiation  that  a  computerized  model 
within  its  domain  of  applicability  possesses  a  satisfactory  range  of  accuracy  consistent  with  the  in¬ 
tended  application  of  the  model”  [47] .  Similarly,  “validation  is  concerned  with  determining  whether 
the  conceptual  simulation  model  (as  opposed  to  the  computer  program)  is  an  accurate  represen¬ 
tation  of  the  system  under  study.  If  a  model  is  valid,  then  the  decisions  made  with  the  model 
should  be  similar  to  those  that  would  be  made  by  physically  experimenting  with  the  system  (if  this 
were  possible)”  [22].  On  the  other  hand,  verification  ensures  that  the  valid  conceptual  model  is 
correctly  translated  into  a  computer  program  that  performs  as  intended  [32].  A  related  term  often 
encoimtered  in  verification  and  validation  discussions  is  model  credibility.  A  model  is  considered 
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credible  if  the  using  decision-maker  accepts  the  model  as  valid  and  uses  it  to  make  and  implement 
decisions  [15]. 

There  are  many  important  issues  to  consider  when  performing  simulation  model  verification 
and  validation.  We  describe  several  of  these  issues  that  should  be  considered  when  we  develop 
the  surrogate  search  validation  process  in  the  following  section.  The  first  consideration  is  that  a 
simulation  model  should  be  developed  and  validated  for  a  specific  purpose  [32,45].  The  purpose 
could  be  to  answer  a  single  or  several  questions  and  each  of  these  questions  must  be  addressed 
separately.  Analysts  should  also  realize  that  a  simulation  model  is  only  an  approximation  of  an 
actual  system  and  can  never  be  made  absolutely  valid  [32].  A  model  can  be  made  more  valid  by 
spending  more  time  and  money  on  the  validation  process,  however  the  analyst  should  consider  the 
cost-effectiveness  of  the  additional  expenditures  required  to  increase  model  validity.  Again,  the 
issue  of  model  applicability  is  the  primary  consideration  when  attempting  to  assess  when  a  model 
is  ‘S^d  enough”  [45].  Finally,  model  verification  and  validation  should  be  an  integral  part  of  the 
model  development  process  [15,32,45]. 

The  rest  of  this  section  describes  the  simplified  verification  and  validation  process  presented 
by  Sargent  [45].  The  topics  addressed  include  descriptions  of  an  integrated  validation  process,  data 
validity,  conceptual  model  validity,  computerized  model  validity,  and  finally  operational  validity. 

6,2,1  Validation  Process,  Sargent  states  that  there  are  three  basic  approaches  to  deter¬ 
mining  simulation  model  validity  [45].  All  three  approaches  require  that  the  model  development 
team  perform  verification  and  validation  during  the  model  development  process.  The  model  de¬ 
velopment  team  makes  the  model  validity  decision  in  the  first  approach  using  a  subjective  method 
based  on  the  results  of  numerous  tests  and  evaluations  performed  during  model  development.  To 
conduct  the  next  approach,  an  independent  verification  and  validation  (IV&V),  a  third  party  in¬ 
dependent  of  the  model  development  team  and  the  end-user  of  the  model  decides  if  the  model  is 
valid.  This  is  accomplished  after  the  model  has  been  developed.  The  independent  party  makes 
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a  subjective  decision  based  on  their  own  evaluation  of  the  delivered  model.  A  scoring  model  is 
used  for  the  final  verification  and  validation  technique  (see  Balci  [4]).  In  this  method,  scores  are 
determined  subjectively  for  various  aspects  of  the  validation  process,  which  are  combined  in  an 
overall  score  for  the  simulation  model.  The  model  is  then  considered  valid  if  the  score  exceeds 
some  pre-determined  score.  Sargent  [45]  notes  that  the  scoring  method  is  rarely  used  in  practice 
and  discourages  its  use  for  determining  model  validity.  He  states  that  the  subjectiveness  of  this 
approach  is  often  masked,  making  the  method  appear  to  be  objective.  He  goes  on  to  describe 
several  other  technical  deficiencies  of  the  method  as  well. 

Both  Sargent  [45]  and  Law  and  Kelton  [32],  as  well  as  other  researchers,  make  it  clear  that 
model  verification  and  validation  should  be  an  integral  part  of  model  development.  To  understand 
that  relationship,  we  now  describe  the  simplified  modeling  process  presented  by  Sargent  [45].  A 
diagram  of  the  modeling  process  appears  in  Figure  6.1.  The  problem  entity  is  the  system  (idea, 
situation,  etc.)  that  is  to  be  modeled.  The  problem  entity  can  either  currently  exist  or  be  a  new 
proposed  entity  that  doesn’t  actually  exist  in  the  real  world.  The  mathematical/logical/ verbal 
representation  of  the  problem  entity  is  called  the  conceptual  model,  and  the  computerized  model  is 
the  computerized  implementation  of  the  conceptual  model.  The  dashed  lines  that  connect  each  item 
are  the  phases  of  the  modeling  development  process.  The  conceptual  model  is  developed  during  the 
analysis  and  modeling  phase,  the  conceptual  model  is  translated  into  a  computerized  model  during 
the  computer  programming  and  implementation  phase,  and  finally  experiments  are  performed  using 
the  computerized  model  during  the  experimentation  phase  in  order  to  make  inferences  about  the 
problem  entity. 

Figure  6.2  illustrates  how  the  verification  and  validation  process  is  integrated  into  the  model 
development  process  [45].  Conceptual  model  validity  is  established  by  examining  the  theories  and 
assumptions  that  are  used  to  develop  the  conceptual  model.  This  results  in  a  conceptual  model  that 
is  a  valid  representation  of  the  problem  entity  for  the  particular  set  of  questions  being  asked.  Once 
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Figure  6.1  Simplified  simulation  model  development  process.  Adapted  firom  Sargent  [45]. 

this  is  established,  computerized  model  verification  is  performed  to  ensure  that  the  computer  code 
correctly  implements  the  conceptual  model.  Operational  validity  is  establishing  the  computerized 
model  output  accurately  represents  the  problem  entity  for  the  region  of  intended  purpose.  Finally 
data  validity  is  defined  as  determining  that  the  data  required  to  accomplish  each  of  these  tasks  is 
correct  and  adequate.  Sargent  [45]  points  out  that  model  development  is  an  iterative  process  and 
that  each  of  the  verification  and  validation  steps  must  be  performed  for  each  iteration. 

Numerous  validation  techniques  are  available  to  the  analyst,  however  no  algorithm  or  theorem 
exists  for  determining  the  best  technique  for  any  given  situation  [45].  Some  of  these  techniques  are 
described  below  with  some  of  the  attributes  that  effect  their  utility.  See  for  example  Sargent  [45] 
and  Balci  [5]  for  a  more  complete  list. 

Comparison  to  Other  Models  The  simidation  model  output  is  compared  to  the  output  from 
another  valid  model.  These  other  valid  models  could  be  analytical  models  or  some  other  valid 
and  credible  simulation  model. 
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Verification 


Figure  6.2  Integrated  simulation  model  verification  and  validation  process.  Prom  Sargent  [45]. 


Pace  Validity  This  technique  involves  asking  system  experts  if  the  model  and/or  its  behavior  is 
reasonable. 

Historical  Data  Validation  Given  that  historical  data  exists,  part  of  the  data  can  be  used  to 
develop  the  model  and  the  remainder  of  the  data  can  be  used  to  test  the  validity  of  the  model. 

Turing  Test  Experts  examine  output  from  the  model  and  problem  entity  and  try  to  determine 
the  source.  The  better  the  model,  the  more  difficult  it  is  for  the  experts  to  differentiate  the 
output. 

Predictive  Validation  System  behavior  predictions  made  by  the  model  are  compared  to  actual 
system  behavior.  The  system  data  could  come  from  an  operational  system  or  experiments 
made  on  the  system. 

The  remainder  of  this  section  examines  the  verification  and  validation  steps  in  more  detail. 
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6.2.2  Data  Validity.  Valid  data  is  required  to  construct  the  conceptual  model,  validate 
the  computerized  model  and  to  perform  experiments  on  the  validated  model.  Model  validation 
is  concerned  with  the  first  two  categories  of  data.  Sargent  [45]  points  out  that  data  vahdity  in 
normally  not  considered  part  of  the  model  validation  process.  However,  valid  data  is  vital  to  the 
construction  of  a  valid  model.  The  problem  is  that  obtaining  enough  vahd  data  is  a  difficult,  time 
consuming  task  and  the  lack  of  it  is  a  major  contributor  to  failed  model  building  attempts. 

6.2.3  Conceptual  Model  Validation.  Conceptual  model  validation  is  achieved  by  ensuring 
that  the  theories  and  assumptions  used  to  create  the  model  are  correct  and  that  the  representation 
of  the  problem  entity,  including  the  model’s  structure,  logic  and  mathematical  relationships,  are 
“reasonable”  for  the  intended  purpose  of  the  model.  Sargent  [45]  suggests  that  face  validation  is 
one  of  the  primary  validation  techniques  used  for  conceptual  model  validation.  Using  flowcharts, 
graphical  models,  and  the  set  of  mathematical  equations  used  in  the  model,  the  analysts  and  system 
experts  can  determine  if  the  conceptual  model  possess  face  validity  for  its  intended  purpose. 

6.2.4  Computerized  Model  Verification.  Computerized  model  verification  ensures  that 
the  computerized  simulation  model  is  performing  as  expected.  This  can  be  an  arduous  and  difficult 
task,  particularly  for  complex  models.  Many  techniques  are  available  to  the  development  team  to 
ensure  that  the  conceptual  model  has  been  correctly  translated  into  computer  code  and  that  the 
code  is  firee  of  bugs.  However,  since  the  primary  purpose  of  this  exposition  is  to  establish  a  frame  of 
reference  for  establishing  a  surrogate  search  validation  process,  these  methods  will  not  be  discussed 
here.  Suffice  to  say,  that  computerized  model  verification  is  a  continuous  process  that  should  occur 
throughout  the  model  development  process  [45]. 

6.2.5  Operational  Validity.  The  final  step  in  the  verification  and  validation  process  is 
operational  validity.  This  process  examines  the  computerized  model’s  output  to  determine  if  it  is 
accurate  enough  for  its  intended  purpose.  Although  this  is  the  last  step  before  declaring  a  model 
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valid,  if  operational  validity  is  not  achieved  the  analyst  must  return  to  one  of  the  previous  steps 
to  determine  the  cause.  Inaccurate  output  is  caused  by  any  combination  (singly  or  together)  of  an 
invalid  conceptual  model,  incorrectly  programmed  computerized  model,  or  invalid  data  [45]. 

Any  of  the  validation  techniques  mentioned  in  Sargent  [45]  and  Balci  [5]  are  applicable  to 
operational  validity.  The  analyst,  system  experts,  and  end-user  determine  the  choice  of  validation 
technique  and  whether  the  evaluations  are  subjective  or  objective.  The  key  to  operational  validity 
is  whether  or  not  output  data  can  be  collected  from  the  problem  entity  itself.  K  the  output  data 
is  available,  the  computerized  model’s  behavior  should  be  compared  to  the  problem  entity.  If  the 
system  is  non-observable,  Sargent  [45]  suggests  comparing  the  model’s  output  to  other  validated 
models. 

At  least  two  different  sets  of  experimental  inputs  are  required  to  ensure  a  hi^  degree  of  model 
validity  according  to  Sargent  [45].  The  outputs  at  each  of  the  different  input  sets  are  compared 
using  either  graphs,  confidence  intervals,  or  hypothesis  tests.  Graphs  are  the  most  commonly  used 
approach  followed  by  confidence  intervals  [45]. 

Graphs  can  be  used  to  examine  the  accuracy  of  a  simulation  model’s  accuracy  over  a  range 
of  different  input  settings.  Types  of  applicable  graphs  include  histograms,  box  plots,  and  behavior 
graphs.  Behavior  graphs  are  simply  graphs  of  output  data  from  the  simulation  model  and  problem 
entity  over  a  range  of  inputs.  It  is  vital  that  the  outputs  graphed  relate  to  the  intended  purpose  of 
the  simulation  model.  The  graphs  can  be  subjectively  examined  by  the  development  team,  system 
experts,  or  as  a  Turing  test  to  determine  operational  validity. 

Confidence  intervals,  simultaneous  confidence  intervals,  and  joint  confidence  regions  are  also 
useful  validation  tools.  They  can  be  formed  for  the  difference  between  the  parameters  of  the  model 
and  system  or  for  the  distribution  of  the  output  values  of  each  over  the  range  of  model  applicability. 
Validation  decisions  can  then  be  made  based  on  the  model  accuracy  observed. 
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Hypothesis  tests  can  also  be  used  to  confirm  operational  validity.  Given  an  appropriate  set 
of  data,  the  hypothesis  to  be  tested  can  be  stated  as  [45]: 

Ho:  Model  is  valid  for  the  acceptable  range  of  accuracy  under  the  set  of 
experimental  conditions 

(6.1) 

Hi  :  Model  is  invalid  for  the  acceptable  range  of  accuracy  under  the  set  of 
experimental  conditions 

Detailed  procedures  for  the  hypothesis  test  method  of  validating  simulation  models  can  be  found 
in  Balci  and  Sargent  [6]  and  Banks,  Carson,  and  Nelson  [7]. 

6,2.6  Verification  and  Validation  Summary.  Simulation  model  verification  and  validation 
is  a  difficult,  yet  vital  task  that  should  be  integral  to  the  model  development  process.  Further, 
since  simulation  models  are  often  used  more  than  once  in  order  to  investigate  problems  other 
than  the  initial  problem  entity,  model  validation  should  be  performed  throughout  the  model  life 
cycle  [5,45].  The  verification  and  validation  task  is  made  difficult  by  the  lack  of  specific  algorithms 
or  theorems  that  specify  the  best  validation  method.  Indeed,  validation  can  often  be  a  subjective 
judgement  made  with  little  or  no  data  firom  a  real-world  problem  entity.  However,  by  following 
the  guidehnes  sketched  above,  the  validation  decision  will  provide  the  user  and  analyst  with  the 
guidance  necessary  for  appropriate  model  use. 

6,3  Surrogate  Search  Validation 

Significant  variance  reduction  achieved  using  the  ACV  method  indicates  that  the  analytical 
model  output  is  correlated  to  the  simulation  model  output  when  the  inputs  to  both  models  are 
similar.  While  this  suggests  that  the  analytical  model  might  be  used  as  a  surrogate  to  the  simulation 
model,  this  is  not  necessarily  the  case.  For  example,  the  two  model  outputs  could  be  negatively 
correlated.  Or,  even  though  variance  reduction  for  the  ACV  method  is  a  result  of  linear  correlation. 
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the  actual  correlation  between  the  two  models  may  more  accurately  be  of  a  quadratic  or  higher 
order  correlation.  Even  if  the  correlation  is  positive  and  linear,  variance  reduction  occurs  in  small 
neighborhoods  about  each  of  the  design  points  replicated.  This  tells  us  little  about  the  behavior 
of  the  two  models  over  the  entire  design  region.  Despite  these  shortcomings,  the  ACV  method  can 
be  used  as  an  integral  part  of  a  formal  process  for  justifying  (validating)  the  use  of  an  analytical 
model  as  a  surrogate  of  the  simulation  model.  In  fact,  the  validation  process  can  be  integrated  into 
a  RSM  simulation  study  using  the  ACV  method  in  a  manner  that  is  nearly  transparent. 

We  develop  a  validation  process  of  an  analytical  model  for  the  purposes  of  conducting  a 
surrogate  search  in  the  context  of  an  RSM  simulation  study  that  uses  the  ACV  method  as  a  primary 
validation  tool.  The  process  is  adapted  from  the  simulation  model  verification  and  validation 
process  described  above  in  Section  6.2.  The  primary  difference  is  that  in  the  above  procedure,  the 
simulation  model  is  validated  against  some  “real-world”  system,  while  in  this  case  the  analytical 
model  is  validated  against  a  previously  validated  simulation  model,  not  the  system  under  study. 
When  constructing  an  analytical  model  in  order  to  generate  an  ACV  this  is  exactly  the  frame  of 
reference.  While  it  may  be  true  that  the  analytical  model  is  (or  can  be  made)  valid  for  the  system 
under  study,  we  are  really  only  concerned  with  comparing  the  output  of  the  analytical  model  with 
the  output  of  the  simulation  model,  not  the  real  system. 

The  surrogate  search  modeling  process  is  illustrated  in  Figure  6.3.  The  figure  is  essentially  the 
same  as  Figure  6.2.  The  main  differences  are  that  a  “Valid/Credible  Simulation  Model  &  Problem 
Entity”  replaces  the  “Problem  Entity”  in  the  surrogate  search  modeling  figure  and  “Operational 
Validity”  is  replaced  by  “Surrogate  Search  Operational  Validity”.  Further,  we  explicitly  state  that 
the  model  we  are  developing  and  validating  is  an  analytical  model.  These  changes  reflect  the  fact 
that  we  are  developing  an  analytical  model  to  serve  as  a  surrogate  of  the  simulation  model,  not  the 
system  under  study.  Otherwise,  the  figure,  and  the  process,  is  identical  to  that  for  the  simulation 
modeling  process. 
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Analytical  Model 
Verification 


Figure  6.3  Surrogate  search  verification  and  validation  process.  Adapted  from  Sargent  [45]. 


Recalling  that  our  overall  goal  is  to  reduce  simulation  study  times,  we  propose  a  surrogate 
search  validation  method  that  is  as  streamlined  as  possible,  while  still  adhering  to  the  vahdation 
principles  outlined  above.  This  is  primarily  accomplished  by  analyzing  the  results  of  the  ACV 
method  and  the  response  surfaces  generated  during  the  initial  stages  of  a  RSM  simulation  study. 
The  validation  process  is  outlined  below.  We  begin  with  a  discussion  of  conceptual  analytical  model 
validity.  Here  we  consider  the  particular  modeling  requirements  necessary  for  an  analytical  model 
to  serve  as  a  surrogate  for  a  particular  simulation  study.  We  then  briefiy  discuss  computerized 
analytical  verification.  Next  we  describe  a  means  of  achieving  surrogate  search  operational  validity 
based  on  the  results  of  the  ACV  method  and  response  surface  comparisons.  This  corresponds  to 
achieving  operational  validity  in  the  simulation  validation  method  above.  We  do  not  address  data 
validity  for  the  surrogate  search  method  other  than  to  say  it  consists  of  ensuring  that  the  valid 
data  used  and  produced  by  the  simulation  model  are  correctly  collected  and  interpreted  for  use  in 
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validating  the  anal3rtical  model.  This  is  relatively  easy  when  compared  to  ensuring  data  validity 
for  the  simulation  model,  however  it  is  a  vital  task  that  must  not  be  ignored.  We  complete  the 
discussion  with  a  brief  summary. 

6,3.1  Conceptual  Analytical  Model  Validity.  Recall  that  conceptual  model  validity  is 
achieved  by  ensuring  that  the  theories  and  assmnptions  used  to  create  the  model  are  correct  and  that 
the  representation  of  the  problem  entity,  including  the  model’s  structme,  logic  and  mathematical 
relationships,  are  “reasonable”  for  the  intended  purpose  of  the  model  [45].  We  achieve  conceptual 
analytical  model  validity  in  the  same  manner,  keeping  in  mind  our  specific  intended  purpose  of 
serving  as  a  surrogate  of  the  simulation  model  for  a  specific  study. 

We  first  consider  the  theoretical  structure  and  assumptions  of  the  analytical  model.  Since  we 
are  developing  a  model  of  an  existing  valid/credible  simulation  model,  this  task  can  be  somewhat 
simpler  than  attempting  to  create  a  model  of  a  real  world  system.  We  already  know  (or  can  reaxlily 
determine)  the  inputs,  parameters,  and  logic  that  are  used  in  the  simulation  model.  For  example, 
in  Chapter  V  it  is  apparent  that  the  conceptual  model  of  the  Psuedo-BRACE  simulation  model 
is  an  open  network  of  queues  with  specific  service  times  and  disciplines.  The  obvious  difiiculty 
in  translating  Psuedo-BRACE  into  an  analytical  model  is  that  the  defined  open  queuing  network 
cannot  be  analytically  solved.  This  is  true  with  most  simulation  models.  If  they  can  be  solved 
analytically  there  is  little,  if  any,  reason  to  construct  the  simulation  model  in  the  first  place.  The 
task  then  is  to  develop  an  anal3diical  model  that  adequately  approximates  the  behavior  of  the 
simulation  model.  This  mirrors  the  task  of  developing  a  conceptual  simulation  model  of  a  real 
system,  since  a  simulation  model  can  at  best  be  an  approximation  of  the  actual  system  [32]. 

As  with  simulation  conceptual  model  validity,  the  technique  recommended  for  evaluating  the 
theories  and  assumptions  used  to  develop  the  conceptual  analytical  model  is  that  of  face  validity  [45]. 
In  this  case,  the  system  expert  wiU  usually  be  the  simulation  analyst  that  regularly  uses  the 
simulation  model  and/or  is  tasked  to  perform  the  specific  simulation  study.  For  analytical  model 
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reasonableness,  some  assumptions  must  be  made  in  order  to  maintain  a  model  that  can  be  solved 
analytically.  For  example,  in  a  simulation  model  a  particular  service  time  may  be  modeled  as 
being  distributed  with  a  WeibuU  distribution.  To  maintain  tractability,  the  analytical  model  may 
have  to  model  that  same  service  time  with  an  exponential  distribution.  These  difFerences  must  be 
considered  when  detennining  the  face  validity  of  the  analytical  conceptual  model.  The  analyst  must 
remember  that  we  are  attempting  to  approximate  the  simulation  model,  not  solve  it.  Additionally, 
the  computational  effort  required  to  compute  the  analytical  model  should  be  considered  as  well. 
The  whole  point  of  this  exercise  is  to  develop  a  surrogate  of  the  simulation  model  that  can  be 
rapidly  evaluated  in  order  to  save  time. 

We  now  turn  to  the  specific  intended  purpose  of  the  analytical  model — ^to  serve  as  a  surrogate 
of  the  simulation  model  for  a  specific  simulation  study.  To  perform  as  a  surrogate  it  must  meet 
two  requirements: 

1.  The  conceptual  analytical  model  must  have  an  output  (s)  that  corresponds  to  the  output  (s) 
of  interest  for  the  simulation  study. 

2.  The  conceptual  analytical  model  must  be  able  to  adjust  the  factors  that  are  adjusted  by  the 
simulation  model  during  a  simulation  study. 

Hence  there  are  specific  mapping  requirements  put  on  some  of  the  outputs  and  mputs  of  the 
conceptual  analytical  model,  based  on  the  specific  simulation  study  to  be  performed.  The  following 
discussion  describes  in  detail  the  mappings  between  the  two  models  that  must  occur. 

Recall  from  Chapter  HI  that  a  simulation  model  can  be  represented  as  the  function  for 
replication  i  as 

f  Vi)  =  f  {<l>^  9{0%)  =  Yf  (6.2) 
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where  is  the  vector  of  structural  parameters,  0^  is  the  vector  of  random  variate  parameters,  g{-) 
is  the  random  variate  generator,  and  g{0^)i  =  Vi  is  the  vector  of  random  variate  processes  that 
drive  the  simulation  model  for  replication  i.  The  output  vector  Yf  consists  of  a  vector  of  stochastic 
processes  realized  dmring  the  ith  rephcation.  The  processes  include  the  performance  measure  of 
interest  for  the  particular  simulation  study  as  well  as  the  realized  “input”  processes.  As  before,  let 
Yf  represent  the  output  stochastic  process  of  interest  with  Y/  the  appropriate  statistical  estimate 
of  the  performance  measure  formed  upon  completion  of  the  planned  replications. 

Tti  a  similar  manner,  the  analytical  model  can  be  represented  as  the  function 

=  (6.3) 

where  is  the  structural  parameter  vector  and  0^  is  the  random  variable  parameter  vector  of  the 
analytical  model.  These  vectors  are  defined  in  a  manner  similar  to  that  for  the  simulation  model. 
The  structural  parameter  vector,  consists  of  those  parameters  that  define  the  structure  of  the 
analytical  model  and  the  random  variable  parameter  vector  9^  consists  of  the  parameters  of  the 
random  variables  that  are  modeled  in  the  analytical  model.  For  the  analytical  model,  there  is  no 
random  number  generator,  so  unlike  a  simulation  model,  for  a  given  vector  0^  an  analytical  model 
will  produce  a  fixed  output.  For  queueing  network  models,  for  example,  the  output  vector  will 
consist  of  state  probabilities  and/or  mean  performance  values. 

The  first  surrogate  requirement  for  the  conceptual  anal5rt;ical  model  states  that  one  of  the 
elements  of  the  output  vector  must  correspond  to  the  simulation  performance  measure,  E\Yp], 
We  refer  to  this  surrogate  output,  if  it  exists,  as  Z,  since  it  is  also  the  same  analytical  model  output 
that  serves  as  the  ACV.  By  “correspond”,  we  mean  Z  and  Yp  are  measures  of  the  same  phenomena 
as  estimated  by  their  respective  models.  Examples  coidd  be  mean  cargo  loaded  per  day  or  mean 
aircraft  throughput  per  hour.  The  difference  is  that  Yp  approximates  the  value  of  the  phenomena 
in  an  actual  system,  while  Z  approximates  the  value  of  for  a  valid  surrogate  model. 
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Implicit  in  the  construction  of  the  conceptual  analytical  model  is  that  a  there  is  a  projection 
and  transformation  of  elements  in  (jy^  and  0^  to  elements  in  and  0^.  It  is  not  expected  that  all 
elements  of  (jy^  and  6^  be  transformed  to  elements  in  and  0^.  For  that  matter,  it  is  not  required, 
or  expected  that  all  elements  of  (fy^  and  0^  are  the  direct  result  of  a  transformation  of  elements 
in  (fy^  and  0^.  Obviously,  the  goal  is  to  project  and  transform  as  many  parameters  as  possible 
between  the  two  models.  The  more  transformations,  the  more  likely  that  the  conceptual  analytical 
model  is  a  valid  representation  of  the  simulation  model.  The  actual  number,  or  portion,  of  elements 
that  must  be  projected  for  to  mimic  is  indeterminate  and  depends  not  only  on  the  models, 
but  the  particular  study  being  performed.  The  best  one  can  say  is  that  an  “adequate”  number 
of  parameters  must  be  mapped  as  determined  by  the  validation  process.  In  previous  chapters,  it 
turns  out  that  for  the  purposes  of  generating  an  ACV,  adequacy  can  be  defined  in  terms  of  the 
amount  of  variance  reduction  achieved.  An  adequate  map  for  generating  an  ACV  could  possibly 
serve  well  for  to  act  as  a  surrogate  of  However,  to  meet  the  second  surrogate  requirement 
above,  specific  elements  of  <fy^  and  0^  must  be  projected  to  0^  and  0^.  These  elements  are  the 
structural  or  random  variate  parameters  that  are  to  be  varied  in  the  particular  simulation  study 
being  performed.  In  a  design  of  experiment  context,  these  elements  are  referred  to  as  treatments 
or  factors.  When  performing  a  search  of  the  experimental  design  space,  these  are  the  inputs  to 
the  simulation  model  that  axe  varied.  If  there  is  no  similar  (surrogate)  parameter  in  the  analytical 
model,  can  hardly  be  used  as  a  surrogate  of  the  simulation  model  for  that  specific  study. 

This  treatment  projection  and  transformation  requirement  can  be  posed  in  the  following  man¬ 
ner.  Consider  a  simulation  study  where  the  measure  of  performance  is  estimated  by  1^.  Let  the 
treatments  that  are  to  be  varied  in  the  study  be  represented  by  —  (Af,Af,...  ,Af)  where 
Af  G  ^0^,  0^^  for  i  =  1, 2, . . .  ,  fc.  So,  for  to  be  a  surrogate  of  /^,  there  must  be  a  function,  g, 
that  projects  and  transforms  A^  to  specific  elements  of  0^  and  0"^,  say  A^  =  (A^ ,  A^, . . .  ,  A^)  that 
corresponds  directly  to  A^.  These  analytical  model  parameters  may  not  be  exactly  the  same  as 
the  ones  in  the  simulation  model.  A  prime  example  is  the  level  of  aggregation  for  the  parameter(s) 
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modeled.  In  any  case,  the  analog  analytical  model  parameter  must  be  adjustable  in  a  manner 
that  corresponds  directly  to  the  adjustments  made  to  the  parameter  (s)  in  the  simulation  model. 
Mathematically,  the  treatment  projection  and  transformation  is  represented  by 

g  (a®)  =  (6.4) 

which  is  the  second  necessary  condition  for  the  conceptual  analytical  model  to  be  a  valid  represen¬ 
tation  of  the  simulation  model. 

Figure  6.4  depicts  the  conceptual  analytical  model  development  and  validation  process  in 
flowchart  form.  As  shown  in  Figure  6.4,  the  process  begins  with  a  valid  simulation  model  and  a 
specific  problem.  In  developing  the  conceptual  model,  the  theories,  assumptions,  and  mathematical 
equations  used  to  construct  it  should  be  considered  reasonable  for  the  purposes  of  approximating 
the  simulation  model.  This  requirement  is  referred  to  face  validity  [45]  and  is  depicted  as  one  of  the 
requirements  in  the  flowchart  decision  block.  Specific  requirements,  or  conditions,  for  conceptual 
analytical  model  validity  are  twofold.  The  first  is  that  there  must  be  an  output  of  the  conceptual 
analytical  model  that  approximates  the  performance  measure  of  interest  of  the  simulation  model. 
The  second  requirement  is  that  a  mapping  from  treatments  of  the  simulation  study  to  the  input 
elements  of  the  conceptual  analytical  model.  These  requirements  also  appear  in  the  flowchart 
decision  block.  Each  of  these  requirements  must  be  met  before  proceeding  on  to  the  next  step 
of  verifying  the  computerized  analytical  model.  If  these  requirements  are  not  met,  the  analyst 
should  determine  if  it  is  cost  efficient  to  pursue  the  ACV  method  before  attempting  to  modify  the 
conceptual  model.  After  all,  the  goal  is  to  reduce  simulation  study  times,  if  possible.  If  the  analyst 
believes  it  is  cost  efficient  to  proceed,  the  conceptual  analytical  model  is  modified  imtil  it  meets 
the  described  requirements. 
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Figure  6.4  Conceptual  anal3rtical  model  development  and  validation  flowchart. 

6,3,2  Computerized  Analytical  Model  Verification.  Verifying  the  computerized  anal3dical 
model  is  essential  to  the  surrogate  search  validation  process.  This  can  often  be  a  much  easier  task 
than  verifying  a  simulation  model.  For  most  analytical  models  software  packages  and/or  well-known 
algorithms  and  mathematical  formulae  exist.  It  is  a  relatively  straightforward  task  then  to  apply 
these  packages  and  algorithms  to  translate  the  conceptual  analytical  model  to  a  computerized 
version.  Some  effort,  such  as  evaluating  test  problems  with  known  answers,  should  be  made  to 
verify  correct  application  of  the  software  and/or  algorithms. 


6,3.3  Surrogate  Search  Operational  Validity.  Surrogate  search  operational  validity  is 
equivalent  to  achieving  operational  validity  for  a  simulation  model.  This  process  examines  the 
computerized  analytical  model’s  output  to  determine  if  it  is  accurate  enough  for  its  intended  purpose 
of  serving  as  a  surrogate  of  the  simulation  model.  Essentially,  when  performing  a  surrogate  search, 
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an  analytical  model  is  attempting  to  predict  the  value  of  the  simulation  model  at  particular  points  in 
the  experimental  design  space.  Therefore,  it  is  logical  to  use  the  predictive  validation  technique  as  a 
means  of  establishing  surrogate  search  operational  validity.  Recall  that  in  the  predictive  validation 
technique,  the  simulation  model  is  used  to  predict  system  behavior  and  then  comparisons  are 
made  between  the  predictions  and  the  actual  observed  values  of  the  system.  In  this  case,  we 
make  predictions  with  the  analytical  model  and  compare  the  predictions  to  the  output  of  the 
simulation  model.  The  comparisons  are  first  made  at  each  design  point  of  an  experimental  design 
by  analyzing  the  results  of  the  ACV  method.  Then  comparisons  are  made  between  the  response 
surfaces  estimated  by  each  model  over  the  experimental  design  space.  If  each  of  these  comparisons 
is  favorable,  surrogate  search  operational  validity  is  established.  Of  course,  the  process  is  iterative 
in  that  modeling  errors  are  addressed  by  modifying  the  conceptual  analytic  model  as  necessary. 
We  describe  each  of  the  validation  steps  below. 

6.3. 3.1  ACV  Design  Point  Validation.  We  wish  to  perform  the  surrogate  search 
operational  validation  in  as  streamlined  a  manner  as  possible  to  maintain  our  goal  of  reducing 
simulation  study  times.  Since  we  are  already  using  the  ACV  method  to  reduce  variance  at  each 
of  the  experimental  design  points,  the  data  necessary  to  validate  the  analytical  model  is  generated 
at  no  additional  cost.  In  a  sense,  we  can  perform  “on-the-fly”  validation  using  the  ACV  results. 
Although  all  design  points  of  the  simulation  study  must  eventually  be  replicated  and  tested  to 
validate  the  surrogate  model,  the  analyst  may  choose  to  perform  the  initial  validation  tests  on  only 
a  few  of  the  design  points  until  he/she  is  fairly  confident  in  the  performance  of  the  analytical  model. 
This  reflects  the  iterative  nature  of  the  validation  process.  In  this  section  we  describe  the  ACV 
method  of  predictive  validation  and  provide  justification  for  its  use. 
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Recall  that  an  ACV  controlled  response  of  the  performance  measure  of  interest,  Y^0),  for 
n  independent  replications  at  a  particular  experimental  design  point  is  defined  by 

Yi0)=Y§{n)-h{Z-iiz)  (6-5) 

where  p  is  found  using  Equation  (2.5).  The  ACV,  Zj  €  Yf  for  replication  j  is  defined  as 

=  y  =  l,2,...,n  (6.6) 

where  ^A{j)  ^  vector  of  sample  means  of  some  subset  of  the  realized  “input”  stochastic  processes 
of  the  simulation  model  observed  during  replication  j.  Therefore  the  stochastic  processes  that  drive 
the  simulation  replication  are  correlated,  by  their  sample  means,  with  the  input  to  the  analytical 
model  during  the  generation  of  an  ACV.  Since  the  variance  of  the  ACV  controlled  response  is  given 
by  [31] 


Var  [y#(^)]  =  (l  -  [v/]  (6.7) 


with 


(J 


2 

z 


(6.8) 


the  amount  of  variance  reduction  achieved  is  a  function  of  the  amount  of  covariance  between  Yp  and 
Z.  Thus,  the  more  the  variance  is  reduced,  the  more  the  outputs  of  the  two  models  axe  correlated 
through  Yf . 

To  achieve  design  point  validity  we  perform  the  ACV  method  at  each  of  the  design  points 
in  the  study’s  design  of  experiments  (DOE).  To  use  the  ACV  results  for  validation  pruposes  we 
must  change  our  frame  of  reference  from  variance  reduction  to  prediction  ability.  Recall  that  if 
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certain  normality  conditions  are  met,  Lavenberg,  Moeller,  and  Welch  [30]  have  shown  that  the  CV 
method  is  the  same  as  a  classical  linear  regression  problem  where  is  the  dependent  variable  and 
Z  —  is  the  independent  variable.  The  linear  regression  model  that  is  estimated  as  the  result  of 
computing  an  ACV  controlled  response  at  a  particular  design  point  is  given  by  [31] 

E[y/]  =  Y§0)  +  HZ-  P-z)  (6-9) 

where  the  ACV  controlled  response,  Y^0)  is  an  estimated  parameter  of  the  model  for  the  given 
replications  (see  Equations  (2.17)  and  (2.18))  and  fiz  is  an  estimate  of  the  constant  fiz-  We  show 
how  analysis  of  this  regression  model,  and  its  parameters,  can  ascertain  the  predictive  ability  of  the 
analytical  model  in  the  neighborhood  of  each  design  point.  In  particular,  if  the  linear  regression 
model  is  adequate,  and  if  ^  1,  and  Yp  Z  then  the  analytical  model  is  an  accurate  predictor  of 

the  simulation  model  in  the  neighborhood  of  that  design  point.  We  examine  each  of  these  conditions 
below. 

When  the  linear  model  estimated  by  Equation  (6.9)  is  found  to  be  adequate  (by  a  statistical 
E-test,  a  large  coefficient  of  simple  determination  (ii^),  and/or  a  small  MSE  for  example)  it  is 
appropriate  to  say  that  E[Yp]  can  be  linearly  predicted  by  the  value  of  Z  (since  fjiz  i®  ^  constant) 
near  the  examined  design  point.  For  each  replication  j,  the  value  of  Zj  is  computed  using  inputs 
that  are  sample  means  of  the  input  stochastic  processes  of  the  simulation  model.  Hence,  when 
both  models  are  given  approximately  the  same  input  settings,  the  value  of  the  simulation  model 
in  the  neighborhood  of  a  design  point  is  accurately  predicted  by  the  linear  regression  function  of  Z 
produced  by  the  ACV  method  at  that  design  point. 

Without  further  analysis,  we  can  only  state  that  we  can  accurately  predict  the  behavior  of  the 
simulation  model  using  a  different  linear  function  of  the  analytical  model  near  each  of  the  separate 
experimental  design  points.  If  ^  ^  1  then  we  know  that  a  unit  change  in  Z  predicts  approximately 
a  unit  change  in  E\Yp]  and  the  value  of  Z  is  simply  a  linear  translation  of  E\Yp],  If  however. 
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Yp  «  Z,  that  translation  is  roughly  equal  to  zero.  If  each  of  the  above  conditions  are  met  at 
each  of  the  design  points,  the  value  of  the  analytical  model  approximately  predicts  the  value  of 
the  simulation  model  in  the  neighborhood  of  each  of  the  design  points.  It  is  up  to  the  analyst  to 
determine  if  the  degree  of  predictive  accuracy  achieved  meets  the  intended  purpose  of  study  and 
the  resulting  surrogate  search.  If  it  has  been  met,  the  analytical  model  is  operationally  vahd  near 
each  of  the  design  points  at  least. 

Simple  methods  to  perform  the  design  point  predictive  validation  foUow.  Obviously,  a  simple 
calculation  can  verify  that  Z  «  Yp.  By  performing  the  ACV  method,  the  other  conditions  can  be 
quickly  checked.  First,  0  is  estimated  during  the  ACV  method  and  can  be  examined  by  the  analyst 
for  closeness  to  the  value  of  one.  One  easy  method  of  quickly  assessing  the  appropriateness  of  the 
suggested  linear  model  is  determining  the  amoimt  of  variance  reduction  achieved  and  examining 
a  two-dimension  scatter  plot  of  Z  versus  Yp.  Since  the  variance  of  the  ACV  controlled  estimate 
of  Yp  is  a  function  of  and  the  coefficient  of  simple  determination  is  the  MLE  of 

the  amount  of  variance  reduction  achieved  is  a  measure  of  model  adequacy.  If  variance  reduction 
achieved  is  relatively  high,  the  appropriateness  of  the  model  can  be  further  verified  by  examining 
the  two-dimensional  scatter  plot  of  Z  and  Yp.  Given  that  the  scatter  plot  indicates  a  linear 
relationship,  combined  with  the  variance  reduction  achieved,  it  can  be  safely  assumed  that  the 
model  appropriateness  is  met.  Further  statistical  testing  can  be  performed,  as  the  analyst  deems 
appropriate. 

When  the  validation  criteria  fails,  the  analyst  has  a  number  of  options.  First,  the  cost 
efficiency  of  attempting  to  modify  the  conceptual  model  in  order  to  develop  a  better  analytical 
model  should  be  considered.  After  a  number  of  failed  attempts,  it  may  make  more  sense  to  abandon 
the  ACV  and  surrogate  search  process  altogether  rather  than  spend  more  time  on  the  analytical 
model.  We  also  point  out  that  if  the  anal3rtical  model  doesn’t  meet  all  of  the  criteria  above,  it  still 
may  serve  as  an  adequate  surrogate  model.  For  example,  ]£ P  ^  1  where  c  is  the  same  constant 
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at  all  of  the  design  points,  a  “new”  analytical  model  can  be  constructed  by  simply  multiplying  the 
output  of  the  original  model  by  1/c.  In  a  similar  manner,  a  constant  translation  of  the  sample 
mean  of  the  analytical  model  from  the  sample  mean  of  the  simulation  model  is  solved  in  a  similar 
manner.  If  predictive  performance  is  poor  at  only  a  few  of  the  design  points,  their  lack  of  accuracy 
may  not  be  a  significant  problem  as  long  as  the  surrogate  search  is  to  be  performed  in  a  direction 
away  from  the  poorly  performing  points.  Results  that  appear  unsatisfactory  may  also  occur  if  the 
performance  measure  has  little  realized  variance  at  some  of  the  design  points.  In  that  case,  little  if 
any  variance  reduction  may  occur  and  the  value  of  ^  may  not  be  close  to  1.  However,  if  the  sample 
means  of  each  model  are  approximately  the  same,  it  is  possible  that  the  analytical  model  is  a  good 
predictor.  Response  smface  comparisons  made  in  the  next  section  may  indicate  that  the  analytical 
model  sufficiently  predicts  the  behavior  of  the  simulation  model  when  considered  over  the  entire 
design  region.  Therefore,  unless  the  ACV  design  point  results  are  highly  inaccurate  it  is  best  to 
proceed  to  the  surface  comparisons  before  making  a  final  decision  on  surrogate  search  operational 
validity. 

6. 3. 3. 2  Response  Surface  Validation.  So  far  we  have  only  discussed  the  operational 
validity  of  the  analytical  model  near  each  of  the  experimental  design  points.  To  achieve  sxurogate 
search  operational  validity  some  means  of  comparing  the  models  over  the  entire  experimental  design 
region  is  required.  One  approach  is  to  predict  the  estimated  simulation  response  surface  with  a 
response  surface  generated  by  the  analytical  model.  This  is  an  obvious  choice  since  it  is  the  next 
step  in  a  RSM  study  following  the  replication  of  the  simulation  model  at  each  of  the  initial  design 
points.  For  illustrative  purposes  we  consider  only  a  first  order  surface  as  estimated  during  the 
initial  stages  of  an  RSM  study.  If  the  analytical  model  response  surface  accurately  predicts  the 
simulation  model  response  surface,  and  the  ACV  design  point  validation  is  successful,  the  analytical 
model  passes  the  predictive  validation  test  and  is  therefore  operationally  valid  as  a  surrogate  of  the 
simulation  model  in  the  prescribed  design  space. 


6-22 


Of  course,  if  the  ACV  design  point  validation  is  met  with  a  high  degree  of  accuracy  at  each 
of  the  design  points,  the  mean  values  of  the  analytical  model  and  the  simulation  model  are  nearly 
the  same  at  the  design  points.  Obviously,  the  response  surfaces  generated  by  each  of  the  models 
will  then  also  be  approximately  the  same  and  surrogate  search  operational  validity  is  achieved. 
However,  if  the  acceptable  bounds  on  the  ACV  design  point  test  are  relatively  large,  the  surface 
comparison  results  are  less  apparent.  Besides  that,  the  simulation  model  response  surface  must 
be  generated  anyway  in  order  to  proceed  on  with  an  RSM  study.  Since  we  have  constructed  the 
analytical  model  with  an  eye  toward  rapid  computation,  the  computation  of  the  analytical  model 
response  surface  and  then  comparing  it  to  the  simulation  response  surface  takes  little  additional 
time.  There  are  many  possible  ways  to  compare  surfaces  and  many  possible  ways  of  constructing 
the  analytical  model  surface.  This  section  describes  one  approach  of  constructing  and  comparing 
surfaces. 

Since  the  surrogate  methods  are  explored  in  the  context  of  an  RSM  model,  the  approach  to 
analytical  model  response  surface  construction  and  response  surface  comparison  is  suggested  by 
RSM.  The  initial  experimental  design  of  an  RSM  model  is  constructed  to  estimate  a  first-order 
empirical  model  in  order  to  define  the  direction  of  steepest  ascent  [12].  The  gradient  vector  of  the 
empirical  model  defines  this  direction.  A  logical  approach  is  to  construct  a  first  order  least  squares 
model  using  the  values  of  the  analytical  model  calculated  at  the  design  points  used  in  the  simulation 
experiments.  The  gradients  of  the  two  surfaces  could  then  be  compared  to  ^e  if  they  point  in  the 
same,  or  nearly  the  same,  direction.  If  the  gradients  are  very  similar,  this  is  an  indication  that 
the  analytical  model  adequately  predicts  the  simulation  model  gradient.  By  also  comparing  the 
response  surfaces  at  the  center  of  the  design  region  (coded  treatments  equal  zero),  the  relative  level 
of  the  responses  are  assessed.  Since  a  first  order  surface  can  be  uniquely  defined  by  its  gradient 
and  response  level  at  the  design  center,  favorable  comparisons  of  both  indicates  that  the  analytical 
model  accurately  predicts  the  behavior  of  the  simulation  model  over  the  entire  design  region  and 
surrogate  search  operational  validation  is  met.  These  comparisons  are  described  below. 
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Consider  an  experimental  design  at  an  initial  stage  of  an  RSM  study.  Let  Yp  be  the  perfor¬ 
mance  measure  of  interest  and  let  Yp  be  a  simulation  model  output  such  that  E[Yp]  =  Yp.  Assume 
that  the  ACV  method  is  used  to  find  a  controlled  estimate  of  For  notational  simplicity,  the 

ACV  terminology  will  not  be  used  in  this  discussion.  The  reader  should  assume  that  all  estimates 
of  simulation  model  responses  are  estimated  using  the  ACV  method.  Now  let  0®  tepresent  the 
coded  treatment  variables  that  will  be  set  in  the  initial  experiments,  where  0®  =  (0f ,  0f , . . .  ,  0f )' 
are  the  k  different  coded  treatments.  Assume  that  the  initial  experimental  design  is  a  2*  factorial 
design  so  that  a  first  order  empirical  model  of  the  form 


Yp  =  6o  +  +  ^*202  +  ■  ■  ■  +  (6.10) 

can  be  estimated.  To  estimate  Equation  (6.10),  n  independent  replications  of  the  simulation  model 
are  generated  at  each  of  the  2'“  design  points.  The  inputs  to  the  simulation  model  at  design  point 
i  are  defined  in  the  following  maimer.  Let  0^^  be  the  vector  of  coded  treatment  levels  at  design 
point  i,  i  =  1, 2, . . .  ,  2^.  These  coded  treatments  axe  mapped  to  the  treatment  factor  vector 
for  design  point  i  by 

Qfi)  Af,)  (6.11) 

such  that 

Afe(</.f,),0f,))  (6.12) 

Then  the  simulation  response  for  the  rephcation  at  the  design  point  is  given  by 

/^(«^fi),p(0fi))i)=l"/(iW  (6-13) 
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Least  squares  analysis  is  applied  to  the  n  x  2^  responses  in  order  to  obtain  the  estimated  parameter 
vector  )'  so  that  the  estimated  response  surface  for  the  simulation  model  is 

given  by 

£[y/]  =  Sf+Sfef +  ---+6fef  (6.14) 

The  usual  statistical  measures  and  tests  can  be  performed  to  determine  the  adequacy  of  the  esti¬ 
mated  model.  The  gradient  of  the  simulation  response  surface,  is  given  by 

=  (6.15) 

and  the  estimated  response  at  the  center  of  the  design  is  bf . 

The  first-order  response  surface  for  the  analytical  model  is  estimated  in  a  similar  manner. 
For  an  analytical  model  that  has  a  validated  conceptual  model,  the  response  measure  of  interest  is 
the  same  as  that  used  for  the  ACV,  Z,  which  estimates  the  same  response  as  Yp .  To  estimate  the 
least  squares  response  surface  for  the  analytical  model,  the  nominal  response  at  each  design  point 
i  is  found  by 

(6.16) 

where 

A(^)G(0(^),E[Yi(,)])  (6.17) 

An  obvious  difficulty  exists  if  some  elements  of  E  are  unknown.  In  that  case  an  estimate 

obtained  from  the  n  replications  performed  at  that  design  point  can  be  used  in  its  place.  The 
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first-order  least  squares  estimate  of  the  analytical  model  response  surface  is  then 


E[Z]  =  b^  +  Sf  01  +  •  •  •  +  b^Qk 


(6.18) 


with  graxiient  =  (Sf  . . .  ,b^y  and  the  estimated  response  at  the  center  of  the  design  is  b^. 
Since  the  first-order  model  is  an  approximation  of  the  analytical  model,  not  a  statistical  estimation, 
there  are  no  statistical  measures  for  testing  the  adequacy  of  the  model.  However  the  first-order 
approximation  can  be  quickly  checked  by  evaluating  the  analytical  model  at  the  center  of  the 
design  and  comparing  the  value  to  If  the  difference  between  the  values  is  small  compared  to 
the  response,  the  approximate  first-order  model  should  be  adequate  for  the  purpose  of  comparing 
gradients. 

The  gradient  of  a  first-order  response  surface  is  constant  over  the  entire  surface.  Therefore  if 
the  gradients  of  the  analytical  and  simulation  response  surfaces  point  in  nearly  the  same  direction, 
a  unit  change  in  any  direction  of  the  inputs  to  each  model  will  result  in  a  similar  change  in  response 
for  both  models  over  the  design  region.  If  the  directions  of  the  two  gradients  are  close  to  each  other, 
the  response  level  changes  experienced  by  changing  the  inputs  to  the  analytical  model  predict  the 
response  level  changes  in  the  simulation  model.  The  gradient  directions  are  compared  by  computing 
the  cos  of  the  angle,  between  the  two  vectors  which  is  given  by 


(6.19) 


where  |a|  is  the  length  of  vector  a.  If  cos  -0  is  close  to  one,  the  two  vectors  point  in  a  similar 
direction.  If  an  analyst  wants  a  more  objective  measure,  a  statistical  test  can  be  performed  on  the 
hypothesis  that  V*^  =  The  test  is  based  on  the  1  —  a  confidence  cone  about  V^.  To  construct 
the  test,  consider  the  semiplane  angle  0i-a  at  the  vertex  of  the  1  —  a  confidence  cone  between  a 
line  on  the  surface  of  the  cone  and  the  axis  of  the  cone,  which  is  defined  by  the  estimated  gradient 
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vector.  Box  and  Draper  [12]  show  that  sin^  Oi-a  is  given  by 


sin^  01-0 


(fc-l)5|sFa(fc-l,U6) 


j=i 


(6.20) 


where  is  the  estimate  of  the  variance  of  any  bj,  and  Vb  is  the  number  of  degrees  of  freedom 
in  the  estimate.  Note  that  S^s  =  for  f  j  =  1, 2, . . .  ,  A:  since  the  coded  treatment  levels  are 

j  i 

used  [12].  Since  cos^  6=1  —  sin^  6,  the  statistical  hypothesis  test  is  posed  as  follows 


Ho: 

Ha:  ^ 


where  the  null  h3rpothesis  is  rejected  if 


COS^'0  = 


|V^ 


>  1  —  sin^9i-o 


(6.21) 


If  it  is  determined  that  the  gradients  are  suflEiciently  close,  the  estimated  responses  at  the 
center  of  each  response  surface  are  also  compared  to  determine  the  analytical  model’s  predictive 
ability.  Given  sufficiently  close  gradients  and  a  small  relative  dffierence  (compared  to  the  magnitude 
of  the  response)  between  the  responses  at  the  center  of  the  design  the  ffist-order  response  surfaces 
for  both  models  are  relatively  the  same.  Hence  a  specific  input  to  both  linear  models  wiU  result  in 
responses  that  are  relatively  the  same.  Given  that  the  first-order  response  surface  approximations 
of  both  models  are  appropriate,  any  particular  set  of  inputs  from  the  entire  design  region  to  the 
analytical  model  wiU  accurately  predict  the  simulation  model  response  given  the  same  inputs  over 
the  entire  region.  Hence,  surrogate  search  operational  validity  is  completely  met. 

As  with  ACV  design  point  validation,  we  again  must  consider  alternative  criteria  when  the 
defined  response  surface  validation  criteria  is  not  met.  In  particular,  if  we  find  that  a  constant 
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translation  of  the  sample  means  of  the  analytical  model  is  required  to  meet  ACV  design  point 
validation,  we  will  need  to  adjust  the  analytical  model  output  at  each  of  the  nominal  settings  in 
order  for  the  design  centers  to  be  relatively  close.  In  a  similar  manner,  if  ^3  c  1  at  all  of  the 
design  points,  the  same  type  of  adjustment  as  used  in  the  ACV  design  point  validation  process 
must  be  made  here  as  well  for  the  two  surfaces  to  be  oriented  in  the  same  directions.  The  point 
is  that  when  we  consider  alternative  criteria  we  are  searching  for  ways  of  adjusting  the  analytical 
model  so  that  we  can  use  it  to  make  accurate  predictions  of  the  simulation  model  output. 

6,3.4  Summary.  We  illustrate  the  surrogate  search  operational  validity  process  in  a 
flowchart  in  Figure  6.5.  The  process  begins  with  a  verified  computerized  analytical  model  and  pro¬ 
ceeds  through  the  two  steps  of  operational  validation,  ACV  design  point  validation  and  response 
surface  validation.  If  both  steps  are  successfully  completed,  the  surrogate  model  is  validated.  If 
some  of  the  ACV  design  point  criteria  are  not,  the  alternate  criteria  described  above  can  be  evalu¬ 
ated  to  determine  if  the  response  surface  validation  criteria  can  be  met.  The  flowchart  highlights 
the  iterative  process  and  the  concern  with  eflficiency.  If  either  step  completely  fails,  the  analyst  must 
decide  if  it  is  more  efficient  to  start  the  validation  process  over  again  by  modifying  the  conceptual 
analytical  model  or  to  abandon  surrogate  search  process. 

Given  that  the  surrogate  search  validation  process  is  successful,  the  analyst  is  confident  that 
the  analjdical  model  is  a  good  surrogate  for  the  simulation  model  over  the  investigated  design 
region.  The  goal,  though,  is  to  use  the  analytical  model  as  a  surrogate  outside  of  the  design  region. 
Certainly  the  successful  validation  process  is  the  best,  and  only,  indication  that  it  will  be  capable  of 
performing  that  task.  The  method  is  grounded  in  accepted  validation  techniques  [45]  and  provides 
an  analyst  with  a  certain  level  of  confidence  in  the  analytical  model.  However,  this  process  provides 
no  guarantee.  This  is  of  course  no  different  than  validating  a  simulation  model.  Simulation  studies 
are  often  performed  because  it  is  difficult,  if  not  impossible  to  perform  experiments  on  the  actual 
system.  Indeed,  the  system  may  only  be  a  proposed  system  that  doesn’t  even  exist  in  the  real 
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world.  Validation  of  the  results  from  these  types  of  experiments  come  only  after  the  expenditure  of 
money  and  time  to  change  or  develop  the  system  as  tested.  For  a  surrogate  search,  we  are  in  a  much 
better  position.  In  this  case,  the  results  of  the  surrogate  search  can  be  validated  by  replicating  the 
simulation  model.  Furthermore,  if  it  turns  out  that  the  surrogate  search  results  are  invalid,  only 
the  relatively  short  time  required  performing  the  surrogate  search  is  lost.  This  ‘Svasted  time”  may 
be  more  than  oflfeet  by  the  time  saved  by  the  successful  application  of  the  ACV  method  of  variance 
reduction. 
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In  summary,  by  following  the  procedures  described  above,  an  analyst  can  efficiently  determine 
the  validity  of  using  a  proposed  analytical  model  as  a  surrogate  of  a  simulation  model.  The  next 
step  then  is  to  actually  perform  the  surrogate  search.  Surrogate  search  procedures  are  outlined  in 
the  next  section. 

Surrogate  Search 

Surrogate  search  methods  could  be  used  in  many  different  situations  in  many  different  types 
of  simulation  studies.  We  choose  to  describe  a  surrogate  search  in  the  context  of  an  RSM  study 
whose  goal  is  to  find  the  Tnaximum  response  for  a  given  set  of  treatments.  RSM  studies  can  be 
formulated  in  many  different  ways.  For  the  purposes  of  this  discussion,  the  initial  stages  of  a  “t5rp- 
ical”  RSM  study  are  described  in  the  following  manner.  At  the  start  of  a  simulation  RSM  study, 
replications  are  performed  according  to  an  e3q)erimental  design  so  that  a  first-order  empirical  model 
can  be  estimated.  Based  on  the  estimated  response  surface,  an  estimated  gradient  is  calculated. 
Replications  are  performed  at  design  points  located  at  multiples  of  the  unit  gradient  vector  posi¬ 
tioned  at  the  center  of  the  original  design.  The  replications  are  performed  to  identify  the  maximum 
response  on  the  steepest  ascent  path.  The  maximum  is  used  to  define  a  new  experimental  design 
center  for  further  experimentation.  A  surrogate  search  procedure  along  the  path  of  steepest  ascent 
is  described  in  this  section. 

Using  the  terminology  described  in  Section  6.3.3.2,  let  Yp  be  the  performance  measure  of 
interest  and  let  Yp  be  an  output  of  simulation  model  /^,  such  that  E[Yp]  =  Yp.  Recall  from 
Section  6.3.3.2  that  the  ACV  method  is  used  to  estimate  the  simulation  response.  The  ACV 
notation  is  not  used  in  order  to  keep  formulas  simple.  Consider  a  2^  factorial  experimental  design 
where  0^  =  (Of,©!,...  are  the  k  different  coded  treatments  that  are  varied  at  the  2^ 

different  design  points.  By  performing  n  independent  replications  of  the  simulation  model  at  each 
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design  point  using  the  appropriate  inputs,  the  estimated  gradient  vector  is  given  by 


(6.22) 


The  task  of  defining  the  new  experimental  design  points  along  the  direction  of  steepest  ascent  is 
simplified  by  finding  the  unit  gradient  vector,  vs,  by 


|V^ 


(6.23) 


In  the  previous  paragraph  it  is  mentioned  that  the  ACV  method  is  used  to  estimate  the  sim¬ 
ulation  responses.  In  order  to  achieve  additional  efficiency,  the  analyst  may  choose  to  approximate 
each  mean  of  the  ACV  at  design  point  i,  using  fewer  random  vector  replications  than 

recommended  in  Chapter  IV.  Since  this  is  the  exploratory  stage  of  investigation,  the  efficiency 
gained  is  warranted.  The  goal  at  this  stage  of  experimentation  is  to  find  a  direction  to  perform 
more  experiments,  not  provide  a  final  answer. 

We  begin  by  describing  the  steepest  ascent  search  method  for  a  simulation  study.  To  perform 
the  steepest  ascent  search  an  appropriate  step  size  must  be  determined.  This  step  size  is  a  function 
of  several  factors,  including  the  resolution  of  the  uncoded  treatment  levels.  For  example,  consider 
the  case  where  one  of  the  treatments  corresponds  to  the  settings  of  a  particular  machine  control. 
If  the  control  can  only  be  positioned  in  a  finite  number  of  specific  settings,  the  coded  step  size 
must  be  adjusted  to  account  for  that.  Without  loss  of  generality,  assume  that  the  coded  step  size 
is  the  length  of  the  coded  unit  vector  |r;^|  =  1.  Then  the  possible  experimental  design  points  can 
be  defined  as  multiples  of  size,  along  the  direction  of  starting  at  the  coded  design  center 
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(0,0, . . .  ,0)'.  Thus  the  steepest  ascent  design  points,  given  by  are  found  by 


ej.,1,  =  lAV 

ef.(2)  = 

:  :  (6.24) 


where  the  total  number  of  design  points  is  determined  by  the  results  of  the  experiments  [12]. 

At  this  point,  the  analyst  is  faced  with  some  important  decisions.  What  is  an  appropriate 
value  for  A^?  Given  that  value  for  A*^,  should  experiments  be  conducted  at  each  integer  multiple 
of  A^  or  can  some  of  the  design  points  be  skipped?  If  the  analyst  is  conservative  and  the  actual 
maximum  is  relatively  far  from  the  design  center,  many  “uninteresting”  time-consuming  replications 
will  be  performed.  Or  the  opposite  might  occur,  where  the  initial  step  is  chosen  as  very  large  and 
the  maximum  is  very  close  to  the  design  center.  Prior  knowledge  of  the  system  or  expert  opinion 
can  guide  the  analyst  in  choosing  the  appropriate  step  size.  This  is  exactly  what  the  analytical 
model  can  provide  when  used  as  a  surrogate  to  the  simulation  model  along  the  direction  of  steepest 
ascent. 

To  perform  a  surrogate  search,  the  analytical  model  is  evaluated  at  a  set  of  design  points  along 
the  path  of  steepest  ascent  in  order  to  identify  the  maximum  response.  Given  that  surrogate  search 
validation  is  achieved,  it  is  likely  that  the  treatment  levels  at  the  design  point  that  corresponds  to 
the  maximum  analytical  model  response  will  be  close  to  the  treatment  levels  that  will  produce  the 
maximum  response  in  the  simulation  model.  Conducting  a  niunber  of  replications  at  or  near  that 
design  point  can  validate  this.  Numerous  methods  of  performing  the  surrogate  search  are  possible. 
We  begin  by  describing  a  method  that  mimics  the  simulation  steepest  ascent  method  described 
above.  Some  alternate  methods  are  sketched  out  at  the  end  of  the  section. 
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Since  the  analytical  model  can  normally  be  evaluated  in  virtually  no  time  at  all  when  com¬ 
pared  to  performing  multiple  replications  of  the  simulation  model,  the  step  size  for  the  surrogate 
search  should  be  relatively  small.  The  actual  size  chosen  can  be  affected  by  the  space  of  permissi¬ 
ble  values  of  each  of  the  treatment  levels.  This  limitation  can  occur  separately  in  each  model  for 
different  treatments,  or  in  both  models  for  the  same  treatment.  When  the  limitation  is  the  same 
in  both  models,  the  analyst  simply  adjusts  the  step  size  as  before.  However,  if  a  treatment  can 
take  on  only  a  finite  number  of  values  in  the  simulation  model,  but  can  take  on  more  values  in 
the  analytical  model,  there  is  little  to  gain  by  evaluating  those  additional  treatment  levels.  After 
all,  the  resultant  response  at  those  levels  can  never  be  validated  by  the  simulation  model.  On 
the  other  hand,  there  may  be  treatments  that  have  a  fewer  number  of  permissible  values  in  the 
analytical  model,  than  the  simulation  model.  In  that  case,  the  analyst  has  no  choice  but  to  adjust 
the  surrogate  search  step  size  based  on  the  treatment  level  limitation.  Nonetheless,  the  surrogate 
search  stiU  provides  the  analyst  information  about  how  the  simulation  model  might  act  along  the 
path  of  steepest  ascent.  Therefore,  based  on  these  treatment  resolution  factors,  the  analyst  should 
choose  a  surrogate  search  step  size,  A^,  as  small  as  practicable. 

Given  a  step  size  of  the  surrogate  search  design  points,  are  given  by 


©«(i)  = 

0^.(2)  = 

0^s(i)  = 


(6.25) 
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where  the  total  number  of  design  points  is  determined  by  the  results  of  the  surrogate  search.  The 
coded  design  points  are  converted  to  the  uncoded  treatment  levels  given  by 

(6.26) 

where  £[Yf„„]  is  the  analytical  model  input  vector  of  expected  values  of  the  means  of  the  random 
processes  of  the  simulation  model  and  is  the  vector  of  structural  parameters  at  each  surrogate 
search  design  point  ss(z).  Given  the  design  points,  the  surrogate  search  is  conducted  by  evaluating 
the  analytic  model  at  each  of  these  points  by 

As  previously  mentioned,  some  of  the  values  of  E  may  not  be  known.  In  that 

case,  estimates  based  on  previous  simulation  replications  can  be  used.  If  the  values  are  subject  to 
unknown  changes  as  the  design  points  move  out  the  path  of  steepest  ascent,  estimates  can  be  made 
by  constructing  an  estimated  response  surface  using  the  data  obtained  in  the  original  experiments. 

The  jSnal  step  in  the  surrogate  search  method  is  to  validate  the  analytical  model  results 
using  the  simulation  model.  Assume  that  the  maximum  response  of  the  analytical  model  on  the 
path  of  steepest  ascent  is  given  by  Zm  which  corresponds  to  surrogate  search  design  point  ss{m). 
A  simple  validation  method  is  accomplished  by  performing  simulation  replications  at  ss{m)  and 
two  sets  of  replications  at  design  points  that  are  a  distance  of  dhAm  from  ss{m).  Any  number  of 
other  validation  methods  are  possible  based  on  the  particular  circumstances  of  the  study.  If  the 


ss(l) 


=  z. 


ss(2) 


=  z. 


ss{i) 


(6.27) 
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simulation  model  doesn’t  validate  the  surrogate  search  results,  the  analyst  simply  designs  a  new 
search  based  on  the  simulation  results  and  his/her  best  judgement. 

The  ability  to  compute  the  analytical  model  rapidly  provides  an  analyst  with  many  options 
for  conducting  a  surrogate  search.  Although,  we  have  developed  a  procedure  for  the  surrogate 
search  method  along  the  path  of  steepest  ascent,  there  is  no  reason  that  the  analyst  should  feel 
constrained  to  that  path  only.  In  fact,  there  are  situations  when  it  is  advantageous  to  conduct 
the  surrogate  search  diflPerently.  For  example,  if  the  resolution  of  one,  or  several,  of  the  treatments 
don’t  allow  for  small  steps  it  may  be  better  to  modify  the  procedure.  In  that  case,  searches 
with  sufficiently  small  step  sizes  can  be  performed  by  holding  the  problem  treatment  constant  and 
varying  the  other  treatments.  Another  strategy  might  be  to  perform  additional  searches  on  paths 
other  than  the  steepest  ascent.  The  point  is  that  the  computational  advantage  of  the  analy1;ical 
model  provides  the  analyst  with  a  means  of  exploring  the  experimental  region  more  thoroughly  than 
possible  with  simulation  model  alone.  The  number  of  possible  surrogate  searches  is  only  limited  by 
the  imagination  of  the  analyst. 

6.5  Summary 

We  have  presented  a  new  method  of  using  an  external  analytical  model  to  reduce  simulation 
study  times  by  employing  the  surrogate  search  procedure.  The  method  is  a  logical  extension  of 
the  ACV  method  of  variance  reduction.  We  have  adapted  the  simulation  model  verification  and 
validation  method  as  a  means  of  justifying  the  surrogate  search  method  for  a  specific  simulation 
study.  Given  a  validated  surrogate  model,  searches  are  performed  by  the  analytical  model  in  order 
to  identify  promising  points  to  perform  simulation  replications.  In  the  next  chapter,  we  demonstrate 
the  effectiveness  of  the  procedure  by  examining  two  different  simulation  models  and  problems. 
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VIL  Application  of  Surrogate  Search  Method 

7.1  Overview 

We  demonstrate  the  eflFectiveness  of  the  surrogate  search  method  developed  in  Chapter  VI 
on  two  different  simulation  models.  The  first  demonstration  consists  of  a  simple  two  factor  RSM 
study  using  the  Psuedo-BRACE  simulation  model  from  Chapter  V.  We  present  this  simple  study 
to  demonstrate  the  basic  apphcation  of  the  surrogate  search  method.  We  follow  that  with  a  RSM 
simulation  study  using  the  USAF  HQ/AMC  Airlift  Flow  Model  (AFM)  simulation  model.  In  this 
case,  we  focus  on  a  “real-world”  size  problem  using  an  actual  validated  and  credible  simulation 
model.  Several  “non-standard”  surrogate  search  issues,  and  their  resolution,  are  examined  in  this 
case  study. 

7. 2  Psuedo-BRA  CE  RSM  Study 

A  simple  RSM  simulation  study  is  conducted  using  Pseudo-BRACE  and  the  analytical  model 
presented  in  Chapter  V  to  demonstrate  the  surrogate  search  method.  We  assume  that  Psuedo- 
BRACE  is  a  valid/credible  simulation  model  for  the  purposes  of  this  example.  We  begin  by  de¬ 
scribing  the  problem  and  the  resulting  RSM  study.  We  then  describe  the  settings  and  output 
statistics  used  m  Pseudo-BRACE  to  conduct  the  study.  This  if  followed  by  a  description  of  the  sur¬ 
rogate  search  validation  process  including  results  from  each  of  the  steps.  Finally  a  surrogate  search 
is  performed  using  the  validated  analytical  model.  The  reader  should  pay  particular  attention  to 
how  the  initial  steps  of  the  RSM  study  are  performed  as  a  result  of  the  surrogate  search  validation 
procedure. 

7.2.1  Study  Description.  The  goal  of  the  RSM  study  is  to  find  the  maximum  steady 
state  mean  for  the  amount  of  cargo,  C,  that  can  be  uploaded  at  a  fictional  airfield  every  24 
hours.  Two  types  of  fictional  aircraft  (C-A  and  C-B)  are  used  to  move  the  cargo,  each  with 
a  different  cargo  capacity.  Both  aircraft  also  have  different  fuel  load  requirements  and  different 
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unscheduled  maintenance  probability  distributions.  The  treatments  that  are  varied  for  the  study 
are  the  proportions  of  each  type  of  aircraft  and  the  overall  arrival  rate  of  aircraft  to  the  airfield. 
The  two  treatments  are  defined  in  the  following  manner.  Let  xi  be  the  proportion  of  arriving 
aircraft  that  are  type  C-A  aircraft,  where  1/6  <  xi  <  1  are  the  possible  values  of  Xi.  Then  1  —  xi 
is  the  proportion  of  arriving  aircraft  that  are  C-B’s.  Let  X2  represent  the  rate  at  which  aircraft  of 
any  type  arrive  at  the  airfield,  in  aircraft  per  hour. 

Since  the  airfield  has  a  limited  number  of  parking  spots,  it  is  expected  that  an  increasing 
number  of  aircraft  will  divert  as  cargo  up-load  is  maximized.  It  is  reasonable  to  expect  that  AMC 
planners  would  want  to  keep  the  number  of  diverting  aircraft  below  a  certain  minimum.  Therefore 
the  maximization  problem  is  constrained  by  a  minimum  number  of  aircraft  diverts  per  aircraft 
arrival.  This  constraint  is  expressed  as  the  probability  that  an  arriving  aircraft  will  divert.  Assume 
that  AMC  wants  P(Divert)  to  be  less  than  0.05. 

The  RSM  maximization  problem  can  then  be  posed  as 


max  C 

s.t.  P(Divert)  <  0.05 


(7.1) 


The  first  step  is  to  construct  the  initial  experimental  design  to  estimate  a  first  order  empirical 
model  using  Pseudo-Brace.  Assume  that  the  AMC  planners  suggest  that  a  suitable  design  center 
is  xi  =  1/2  and  X2  =  1.25  aircraft  per  hour.  Using  that  as  the  center,  a  2^  factorial  design  is 
constructed  in  the  following  manner.  Based  on  the  planners’  suggestions  the  high  and  low  levels 
for  x\  are  x^  =  2/3  and  xf  =  1/3.  For  treatment  two,  the  high  and  low  levels  are  X2  =1.5  and 
X2  =  1.0  aircraft  per  hour.  The  initial  experimental  design  is  listed  in  Table  7.1.  The  levels  of  the 
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coded  treatment  variables,  0  =  (©i ,  ©2)'  axe  found  using  the  following  formulas 


©i 


©^2 


xi-1/2 

1/6 

d  - 1.25 
0.25 


j  =  H,L 


j  =  H,L 


(7.2) 

(7.3) 


The  coded  experimental  design  appears  in  Table  7.2. 


Table  7.1  Initial  RSM  study  2^  factorial  design. 


Design 

Point 

Uncoded 
Treatment  Level 

Xi 

1 

1/3 

1.0 

2 

2/3 

1.0 

3 

1/3 

1.5 

4 

2/3 

1.5 

Table  7.2  Initial  RSM  study  2^  factorial  design. 

Coded 

Design  Treatment  Level 


Point 

©1 

©2 

1 

-1 

-1 

2 

-1-1 

-1 

3 

-1 

-1-1 

4 

+1 

-1-1 

7.2.2  Pseudo-BRACE  Settings.  We  begin  by  describing  the  airfield  resources  and  aircraft 
settings  within  Pseudo-BRACE.  The  airfield  being  simulated  is  essentially  the  same  as  the  one 
studied  in  Chapter  V,  with  one  diflFerence.  There  are  nine  K-loaders  at  this  airfield  versus  the 
eight  assigned  to  the  airfield  in  chapter  V.  The  airfield  resources  are  listed  in  Table  7.3.  Other 
network  settings  include  the  length  of  the  standard  ground  time,  2.25  hours,  and  the  length  of  time 
an  aircraft  will  wait  before  diverting,  which  is  2  hours.  Recall  that  an  aircraft  without  hazardous 
cargo  will  be  parked  at  a  spot  with  a  fuel  hydrant  pit,  if  one  is  available.  If  not,  it  will  be  parked 
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first  at  a  non-hazardous  parking  spot  and  then  a  hazardous  parking  spot.  The  probability  that  an 
arriving  aircraft  will  up-load  hazardous  cargo  is  0.10. 


Table  7.3  RSM  study  airfield  resources. 


Resource 

Number 

Total  Parking  spots 

12 

Hydrant  parking  spots 

4 

Hazardous  parking  spots 

4 

Fuel  Trucks 

8 

Fuel  fill  stand 

1 

K-Loaders 

9 

Forklifts 

19 

Loading  docks 

10 

The  aircraft  attributes  are  provided  in  Table  7.4.  Based  on  the  fuel  load  requirements  and 
fuel  truck  capacities,  fuel  trucks  will  have  to  make  2  trips  to  refuel  a  C-A  aircraft  and  3  trips  to 
refuel  a  C-B  aircraft.  The  cargo  up-load  requirement  and  average  pallet  cargo  weight  results  in  a 
total  of  12  pallets  for  every  C-A  up-load  and  15  pallets  for  every  C-B  up-load.  This  translates  to 
3  K-loaders  (K-loader  capacity  equals  5  pallets)  for  every  aircraft.  The  aircraft  unscheduled  main¬ 
tenance  category  probabilities  are  recorded  in  Table  7.5.  Within  each  category,  the  unscheduled 
maintenance  time  is  uniformly  distributed.  Scheduled  maintenance  (performed  concurrently  with 
unschediiled  maintenance)  is  fixed  at  30  minutes  and  LOX  servicing  is  fixed  at  9  minutes. 


Table  7.4  RSM  study  aircraft  parameters. 


Type 

Probability 

Fuel  load 
(gals) 

Cargo  load 
(tons) 

Fuel  receive 
rate  (gpm) 

C-A 

Xi 

10000 

26.0 

450.0 

C-B 

1  — 

15000 

32.6 

450.0 

Table  7.5  RSM  study  unscheduled  maintenance  probabilities. 


Time  category  (hours) 


Type 

0-4 

4-8 

8-12 

12-16 

16-24 

24-48 

48-72 

None 

C-A 

C-B 

0.000 

0.043 

0.010 

0.057 

0.000 

0.029 

0.000 

0.036 

0.010 

0.021 

0.000 

0.021 

0.010 

0.007 

0.930 

0.786 
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At  each  design  point  of  the  initial  study  shown  in  Table  7.2,  20  replications  of  the  Pseudo- 
BRACE  model  are  generated.  Aircraft  arrive  to  the  airfield  via  a  simulated  Poisson  process  with 
mean  arrival  rate  equal  to  the  appropriate  value  of  X2  every  hour.  One  thousand  aircraft  arrive 
at  the  airfield  during  each  simulation  replication.  To  eliminate  the  effects  of  initial  transient  bias, 
statistics  are  gathered  on  the  last  900  arrivals.  The  reader  should  interpret  all  definitions  of  the 
following  statistics  to  implicitly  include  these  truncations.  The  steady  state  mean  of  cargo  up¬ 
loaded  every  24  hours,  C,  is  estimated  in  the  following  manner.  Let  Cij  be  the  amount  of  cargo 
up-loaded  by  the  aircraft  to  depart  the  airfield  during  simulation  replication  j.  Further  let  hj 
represent  the  total  time  simulated  in  hoius  and  dj  be  the  number  of  aircraft  that  depart  the  airfield 
(non-diverting  aircraft)  during  replication  j.  Then  the  mean  amount  of  cargo  up-loaded  every  24 
hours  for  replication  j  is  given  by 


24 

j  =  l,2,...,20 

^  *=1 


(7.4) 


To  estimate  the  probability  that  an  arriving  aircraft  will  divert,  consider  the  indicator  variable,  Uj, 
defined  as 


f 

1  if  arriving  aircraft  for  rep  j  diverts 

lij  =  < 

0  otherwise 

Then  the  probability  of  diverting,  for  replication  j  is  given  by 

1 

=  j  =  l,2,...,20 

•  1 

''  t=l 


(7.6) 


(7.7) 
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so  that  P(Divert)  is  estimated  by 


j-i 

Each  of  these  responses  are  calculated  at  each  design  point  in  order  to  estimate  the  first  order 
empirical  model. 

Z2.3  Surrogate  Search  Validation  /  Initial  RSM  Results.  We  now  focus  on  the  surrogate 
search  validation  process  and  the  initial  results  of  the  RSM  study.  We  begin  by  establishing 
conceptual  anal5rtical  model  validity  Then  surrogate  search  operational  validity  is  established 
during  the  initial  stage  of  the  RSM  study. 

7.2.3. 1  Conceptual  Analytical  Model  Validity.  Given  a  valid/credible  simulation 
model  and  a  problem  statement  with  known  outputs  and  treatments,  there  are  essentially  two 
requirements  for  conceptual  anal5rtical  model  validity.  The  first  requirement  is  model  face  validity 
and  the  second  requires  the  analytical  model  to  have  outputs  and  treatment  parameter  inputs  that 
correspond  to  the  outputs  and  treatments  of  the  simulation  model.  See  Figure  7.1  for  a  flowchart 
of  the  process.  Thus  far,  we  have  described  the  credible  simulation  model  (Psuedo-BRACE)  and 
identified  the  problem  statement,  output  performance  measures,  and  input  treatments.  We  address 
the  two  conceptual  analytical  model  validity  requirements  below. 

We  begin  by  examining  conceptual  analytical  model  face  validity.  The  proposed  conceptual 
analjdical  model  is  the  single  class  closed  queueing  network  model  described  in  Chapter  V  and 
depicted  in  Figure  5.3,  modified  to  acconunodate  two  classes  (chains)  of  aircraft.  As  before,  we 
solve  the  model  using  the  MVA  algorithm  and  the  fork-join  node  approximation.  To  modify  the 
model  for  two  classes,  the  service  disciplines  at  the  stations  representing  cargo  up-load  (station  4) 
and  refueling  (station  6  and  7)  are  changed  from  first-come  first-served  (FCFS)  to  processor  sharing 
(PS)  service  disciplines.  This  is  necessary  since  all  classes  of  customer  at  FCFS  stations  must  have 
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Figure  7.1  Conceptual  analytical  model  validation  jlowchart. 


the  same  mean  service  time  in  order  to  be  solved  by  the  MVA  algorithm  [8,42].  However,  difiFerent 
classes  of  customers  at  PS  stations  may  have  different  mean  service  times  that  correspond  to  the 
class  of  the  customer.  PS  stations  assume  that  the  station  servers  serve  aU  customers  in  the  station 
in  parallel,  each  customer  receiving  an  equal  share  of  the  service.  This  type  of  service  discipline  is 
also  referred  to  as  time  division. 


The  MVA  algorithm  is  simply  modified  to  account  for  more  than  one  class  of  customer  [13, 16]. 
Essentially  an  additional  loop  is  added  to  the  algorithm  to  account  for  each  different  class  of 
customer.  As  with  the  single  class  MVA  algorithm,  the  response  time  for  each  class  of  customer 
is  calculated  for  population  size  N  =  (iVi,iV2, . . .  where  r  is  the  number  of  classes  in  the 

network,  based  on  the  mean  queue  length  when  one  fewer  customer  is  in  the  system.  The  cycle  time 
for  each  class  of  customer  is  then  calculated  so  that  throughput  and  utilization  for  each  class  of 
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customer  can  be  found.  These  values  are  then  used  to  find  the  mean  queue  length  of  each  customer 
class  at  each  station  for  customer  population  N.  The  fork-join  network  response  times  are  also 
found  based  on  the  population  of  each  customer  class  in  the  same  manner  as  for  one  class. 

The  nominal  service  times  for  each  class  of  customer  and  the  number  of  servers  at  each 
design  point  are  based  on  Pseudo-BRACE.  These  values  are  listed  in  Table  7.6.  The  number  of 
cargo  servers  is  based  on  the  total  number  of  K-loaders  in  Pseudo-BRACE  divided  by  the  number 
of  K-loaders  required  by  each  aircraft.  The  proportion  of  aircraft  refueled  by  hydrant  system,  pk, 
is  also  required  in  order  to  evaluate  the  anal3rt;ical  model.  The  nominal  value  of  this  probabifity  is 
not  known  until  the  simulation  model  is  replicated.  An  estimate  of  pn  =  1/3  can  be  used  based 
on  dividing  the  number  of  parking  spots  with  hydrant  systems,  4,  by  the  total  number  of  parking 
spots,  12.  Or  the  observed  sample  mean  of  this  value  from  the  simulation  replications  can  be  used 
instead. 


Table  7.6  MVA  model  RSM  study  settings. 


Station 

Discipline 

Servers 

Mean  Service 
Times  (hours) 
C-A  C-B 

0 

FCFS 

1 

l/x^ 

l/a;2 

1 

FCFS 

1 

0-033 

0.033 

2 

Delay 

M 

0.167 

0.167 

3 

Delay 

Inf 

1.505 

3.372 

4 

PS 

3 

0.936 

1.117 

5 

Delay 

Inf 

0.160 

0.160 

6 

PS 

1 

0.410 

0.596 

7 

PS 

8 

1.683 

2.762 

8 

Delay 

Inf 

0.167 

0.167 

9 

FCFS 

1 

0.033 

0.033 

Based  on  the  results  of  the  ACV  application  for  the  single  class  version  of  this  model  in 
Chapter  V,  it  is  reasonable  to  expect  the  proposed  conceptual  model  to  adequately  approximate 
Psuedo-BRACE  for  the  problem  described,  subject  to  operational  testing.  There  is  a  certain 
amount  of  uncertainty  concerning  the  use  of  the  PS  service  discipline  at  the  cargo  up-load  and 
refueling  stations.  Obviously  the  service  at  those  queues  in  Pseudo-BRACE  do  not  operate  in  this 
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fashion.  However,  the  goal  is  to  find  an  approximate  solution  that  can  provide  results  based  on  the 
different  aircraft  parameters.  This  assumption  allows  the  model  to  account  for  the  different  service 
time  parameters  of  the  two  different  types  of  aircraft.  Despite  the  uncertainty,  it  is  reasonable 
to  expect  that  the  results  from  these  PS  queues  will  apprcxximate  the  behavior  Psuedo-BRACE 
queues.  Therefore,  at  this  point  of  the  study,  we  believe  the  conceptual  analytical  model  meets  the 
requirements  of  face  validity. 

We  now  determine  if  the  conceptual  analjrtical  model  provides  for  the  necessary  output  and 
treatment  mappings.  First  of  all,  the  analjrtical  model  must  produce  mean  responses  for  the  cargo 
up-loaded  in  24  hours  and  the  probability  of  diverting.  These  responses  can  be  calculated  as 
functions  of  the  calculated  throughput  at  station  0.  Let  A  =  (Ai,  A2)'  be  the  throughput  (aircraft 
per  hour)at  station  0  for  aircraft  classes  C-A  and  C-B  respectively.  Using  the  cargo  capacity  for 
each  type  of  aircraft  given  in  Table  7.4  as  26  and  32.6  tons  for  C-A  and  C-B  aircraft,  the  mean 
number  of  tons  of  cargo  up-loaded  every  24  hours  is  given  by 

7^  =  24[26  32.6]A  (7.9) 

For  an  open  capacitated  queue,  aircraft  will  divert  when  they  arrive  to  an  airfield  that  is 
at  capacity.  (Note,  this  is  different  than  Pseudo-BRACE,  where  aircraft  won’t  divert  until  they 
have  waited  a  total  of  2  hours  for  a  parking  space.)  The  equivalent  closed  queueing  network  is  at 
capacity  when  station  0  is  idle.  The  probability  that  station  0  is  idle  is  given  by  1  —  Uq,  where  Uo 
is  the  station  0  server  utilization  (expected  number  of  servers  busy).  Uo  is  found  by 

l7o  =  soil'A  =  l/a:il'A  (7.10) 
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where  5oi  =  ^02  is  the  mean  service  time  at  station  0  of  type  C-A  and  C-B  aircraft  respectively. 
Thus  the  probability  that  an  arriving  aircraft  will  divert  is  calculated  by  the  analytical  model  as 

=  (7.11) 

To  find  7^  and  for  the  purposes  of  generating  an  ACV,  the  observed  sample  means  of 
the  aircraft  arrival  rate,  unscheduled  maintenance,  cargo  up-load  time,  and  aircraft  refueling  by 
hydrant,  for  every  simulation  replication,  are  used  as  inputs  to  the  analytical  model.  Similarly, 
a  surrogate  search  for  these  output  performance  measures  can  be  performed  by  providing  the 
analytical  model  with  the  appropriate  input  parameters. 

Now  we  describe  how  the  two  treatment  levels  are  adjusted  in  the  conceptual  analytical 
model.  Since  the  Pseudo-BRACE  airfield  has  a  total  of  12  parking  spots,  the  closed  analytical 
model  has  a  total  population  of  12  aircraft.  Thus,  the  aircraft  population  in  the  analytical  model  is 
represented  by  =  {Nf,  N^Y  such  that  +  iV^  =  12.  Let  and  represent  the  number 
of  C-A  aircraft  and  C-B  aircraft  in  the  system  respectively.  Then  the  treatment  corresponding  to 
the  proportion  of  aircraft  is  adjusted  in  the  analytical  model  by  changing  the  totals  of  aircraft  for 
each  class.  For  example,  for  xi  =  1/3,  =  4  and  =  8.  This  is  an  example  of  the  situation 

described  above  when  the  resolution  of  the  two  models  is  not  the  same.  Nonetheless,  the  first 
treatment  level  can  be  adjusted  in  the  analytical  model.  To  adjust  the  aircraft  arrival  rate,  the 
mean  service  time  for  both  types  of  aircraft  at  the  “arrival”  station  0  is  adjusted  appropriately.  The 
mean  service  time  at  station  0  for  this  closed  system  corresponds  to  the  reciprocal  of  the  arrival  rate 
for  an  equivalent  open  capacitated  network.  Hence  for  X2  =  1.5,  let  sqi  =  -§02  =  1/1.5  =  2/3,  where 
sor  is  the  mean  service  time  at  station  0  for  customer  class  r  =  1, 2.  In  this  case,  the  treatment 
levels  have  the  same  resolution  (infinite)  for  both  models.  Thus,  the  initial  experimental  design 
settings  for  the  analytical  model  is  shown  in  Table  7.7. 
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Table  7.7  Analytical  model  settings  for  2^  factorial  design. 


Design 

Point 

Uncoded 
Treatment  Level 

N 

SOr 

1 

[4  8]' 

1.0 

2 

[8  4]' 

1.0 

3 

[4  8]' 

1.5 

4 

[8  4]' 

1.5 

We  have  now  shown  that  conceptual  analytical  model  validity  is  achieved  with  the  proposed 
MVA  model.  The  next  step  is  to  perform  the  initial  design  of  experiment  using  the  ACV  method  in 
order  to  assess  surrogate  search  operational  validity  and  complete  the  initial  stage  of  the  RSM  study. 
We  skip  the  computerized  analytical  model  verification  process  other  than  to  say  the  computer 
implementation  of  the  conceptual  model  was  verified  as  correctly  applied. 

7.2,3.2  Surrogate  Search  Operational  Validity,  Surrogate  search  operational  validity 
is  a  two  step  process  that  is  accomplished  during  the  first  two  steps  of  the  RSM  study.  See  Figure 
7.2.  First,  we  replicate  the  simulation  model  to  estimate  the  output  performance  measures  using 
the  ACV  method.  The  ACV  results  are  analyzed  at  each  design  point  to  assess  the  first  level 
of  operational  validity.  This  is  referred  to  as  ACV  design  point  validation.  Secondly  response 
surfaces,  and  gradients,  are  estimated  using  results  from  both  models.  The  response  surfaces  are 
then  compared  to  judge  the  operational  validity  of  the  analytical  model.  When  each  of  these 
requirements  are  met,  the  anal3d;ical  model  meets  sinrrogate  search  operational  validity. 

To  assess  ACV  design  point  validity  and  to  complete  the  RSM  study,  we  perform  20  replica¬ 
tions  of  the  Pseudo-BRACE  model  at  each  of  the  2^  design  points.  The  ACV  method  is  used  to 
provide  controlled  estimates  of  the  mean  amount  of  tons  of  cargo  up-loaded  every  24  hours, 
and  the  estimated  probability  that  an  aircraft  diverts,  6^,  The  ACV  for  cargo  up-loaded  is  7"^  and 
for  the  probability  of  diverting,  it  is  It  turns  out  that  the  observed  number  of  aircraft  diverting 
at  each  design  point  in  the  initial  study  is  so  low  that  meaningful  comparisons  for  the  probability 
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of  diverting  are  not  useful.  Since  the  probability  of  diverting  as  calculated  by  the  analytical  model 
is  a  linear  function  of  aircraft  throughput,  the  ACV  method  is  used  to  find  controlled  estimates 
of  total  aircraft  throughput  at  each  design  point.  Let  the  estimated  mean  aircraft  throughput  for 
Pseudo-BRACE  be  given  by  L^,  The  ACV  for  throughput  is  represented  by  A,  These  results  are 
also  compared  for  the  purposes  of  judging  ACV  design  point  validity  using  the  following  criteria: 


1.  ACV  linear  regression  model  is  appropriate 
(a)  “Significant”  variance  reduction 
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(b)  Linear  scatter  plot 


2.  1.0 

3.  «  Z 

The  variance  reduction  achieved,  /?,  Vp ,  and  Z  for  each  response  are  provided  in  Table  7.8. 
At  design  point  2,  an  ACV  controlled  response  was  not  calculated  for  probability  of  diverting  since 
no  aircraft  diverted  for  any  of  the  20  repUcations.  Note  that  the  mean  of  each  ACV  is  estimated 
using  only  100  random  vectors  at  each  design  point.  As  mentioned  previously,  since  the  goal  is  to 
find  a  gradient,  not  the  final  answer,  the  time  saved  at  this  level  of  investigation  will  pay  off. 


Table  7.8  ACV  results  at  all  design  points. 


Response 

Variance 
Reduction  (%) 

z 

Cargo 

85.20 

0.928 

P(Divert) 

8.07 

Throughput 

87.83 

0.924 

0.99 

1.00 

Design  Point  2  (HL)  | 

Cargo 

86.61 

0.990 

677.2 

0.00 

1.00 

Design  Point  3  (LH) 

Cargo 

63.76 

0.915 

1073.6 

■ttfAfll 

P(Divert) 

13.15 

0.618 

0.00 

Throughput 

66.23 

0.910 

1.47 

1.49 

Design  Point  4  (HH) 

68.97 

1.038 

■nnuiii 

P(Divert) 

-2.16 

1012!^ 

liTiliM 

■iSiU 

Throughput 

66.69 

Two  dimensional  scatter  plots  for  aircraft  throughput  at  each  design  point  are  provided  in 
Figure  7.3.  Siace  the  scatter  plots  for  cargo  up-loaded  are  essentially  the  same  as  the  throughput 
plots,  they  are  not  provided.  The  scatter  plots  for  P(Divert)  are  not  provided  since  few  diverts 
actually  occurred. 
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Figure  7.3  Two  way  scatter  plots  for  aircraft  throughput  (aircraft/hr). 


Based  on  the  results  presented  in  Table  7.8  and  the  scatter  plots,  the  ACV  design  point 
validation  criteria  are  met  for  the  cargo  up-load  and  throughput  responses.  The  ACV  results  for 
probability  of  diverting  are  essentially  disregarded  due  to  the  small  number  of  diverts  observed. 
However,  the  analytical  model  did  predict  small  diverting  probabilities  at  each  design  point.  Addi¬ 
tionally,  the  probability  of  diverting  is  a  linear  function  of  throughput  in  the  analytical  model,  and 
the  results  for  throughput  are  satisfactory.  Based  on  those  facts,  ACV  design  point  validity  is  met. 

The  simulation  and  anal3dical  model  response  surfaces  are  estimated  in  order  to  verify  the 
validity  criteria  at  the  response  surface  level  and  to  identify  the  steepest  ascent  gradient.  The 
estimated  coefficients  for  the  simidation  and  analytical  models  are  provided  in  Tables  7.9  and  7.10. 
The  estimated  variance  for  the  estimated  simulation  coefficients  are  also  provided.  The  simulation 
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response  surfaces  are  based  on  ACV  controlled  responses  for  cargo  up-loaded  and  throughput.  For 
the  probability  of  diverting  response,  the  estimates  are  based  on  the  uncontrolled  responses  since 
the  ACV  method  could  not  be  applied  at  design  point  3. 


Table  7.9  Simulation  response  surface  parameter  estimates. 


Response 

bi 

bf 

iiiiiEimiii 

Cargo 

866.39 

168.74 

2.0882 

P(Divert) 

■iinTgaaa 

KljiQSlJIII 

Throughput 

■IQHBI 

Table  7.10  Analytical  response  surface  parameter  estimates. 


Response 

b^ 

— 

Cargo 

862.6426 

-29.2648 

165.4341 

P(Divert) 

0.0075 

-0.0028 

0.0066 

Throughput 

1.24 

0.0042 

0.24 

The  criteria  for  response  surface  validity  are  that  the  gradient  vectors  of  each  surface  be 
approximately  equal  and  that  the  response  at  the  center  of  each  surface  be  relatively  the  same 
value.  The  gradient  vectors,  and  are  compared  by  finding  cos^,  where  ip  is  the  angle 
formed  between  the  two  gradients.  cosip  ^  1  the  gradient  vectors  point  in  similar  directions. 
The  gradients  are  also  compared  by  performing  a  statistical  test.  The  test  determines  if  is 
contained  in  the  1  -  a  confidence  cone  about  the  The  test  is  reported  by  listing  the  angle  that 
defines  the  95%  confidence  cone  ('^o.os)*  This  value  can  then  be  compared  to  ip.  The  values  of  ip^ 
cos  Ip ^  ^0.05  responses  at  the  center  of  each  response  surface  are  provided  in  Table  7.11  for 

aU  three  responses. 


Table  7.11  Response  surface  condition  results. 


Response 

Ip 

cosip 

V^0.05 

BIB 

0.98 

0.99 

0.96 

866.4 

862.6 

P(Divert) 

15.79 

23.33 

■IMlf 

■iliM 

Throughput 

1.00 

MKm 
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The  results  of  the  comparisons  listed  in  Table  7.11  indicate  that  the  response  surfaces  compare 
favorably.  Although  ^  for  cargo  up-load  and  throughput  is  larger  than  the  respective  values  of 
Q5,  the  angles  are  still  very  small  as  reflected  by  the  cos^  values.  However,  the  ^q.os  values 
for  these  two  responses  are  very  small  due  to  the  small  estimated  variances  of  the  gradient  vector 
components.  The  value  of  ^  for  probabihty  of  diverting  is  larger  than  the  other  two  angles,  but 
the  cos  is  very  close  to  one  and  is  contained  in  the  95%  confidence  cone  about  Therefore 
it  is  safe  to  conclude  that  the  directions  of  the  simulation  and  analytical  model  gradient  vectors 
point  in  nearly  the  same  direction  for  all  responses.  The  responses  at  the  center  of  each  design  are 
also  very  close  for  the  response  surfaces  generated  by  both  models.  Hence,  the  response  surfaces 
generated  by  both  models  are  indeed  very  similar.  Based  on  the  reported  experimental  results,  the 
conditions  for  response  surface  validity,  are  met.  Both  elements  of  surrogate  search  operational 
validation  have  been  met,  providing  us  with  a  validated  surrogate  search  model. 


7.;?.^  Surrogate  Search  Results.  Given  a  validated  surrogate  search  model,  we  conduct  the 
surrogate  search.  Since  the  amount  of  cargo  up-loaded  is  the  response  that  is  to  be  maximi25ed,  the 
gradient  based  on  the  cargo  up-load  surface  is  used  for  the  steepest  ascent  seardi.  As  mentioned 
previously,  the  gradient  estimated  via  the  simulation  response  surface  will  be  used  to  determine 
the  search  direction.  The  unit  gradient  vector,  for  cargo  up-load  is  given  by 


-0.1897 

0.9818 


(7.12) 


Using  the  surrogate  search  method  outlined  in  Section  6.4,  the  next  step  is  to  determine  an 
appropriate  step  size  for  the  surrogate  search.  The  resolution  of  the  aircraft  proportion  treatment 
level  within  the  analytical  model  and  determines  the  possible  step  sizes.  The  center  of  the 
design  region  corresponds  to  Xi  =  1/2  ^  =  (6,6)'.  The  direction  of  steepest  ascent  points  in 

a  negative  direction  for  aircraft  proportion.  Therefore  each  surrogate  search  step  from  the  design 
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center  will  reduce  the  number  of  C-A  aircraft  in  the  MVA  model  network  by  one.  This  can  continue 
until  =  2  since  the  aircraft  proportions  are  constrained  by  1/6  <  xi  <  1.  At  that  point,  the 
search  can  continue  with  the  same  step  sizes  for  X2  holding  Xi  =  1/6.  Reducing  the  number  of  C-A 
aircraft  by  one  corresponds  to  changing  the  coded  treatment  level  by  -0.5.  The  coded  step  size  is 
then  found  by  solving 

^  =  2.66  (7.13) 

vf 


Based  on  this  step  size,  the  coded  and  uncoded  values  of  the  treatment  levels  for  the  surrogate 
steepest  ascent  search  are  provided  in  Table  7.12.  Note  that  the  first  step  is  skipped  since  it  is 
inside  the  design  region. 


Table  7.12  Surrogate  search  steps. 


Treatment  Levels 


Step 

Coded 

Uncoded 

02 

N 

1/ ^Or 

0 

0.0 

0.0 

[6  6]' 

1.00 

1 

-1.0 

5.18 

[4  8]' 

2.29 

2 

-1.5 

7.76 

[3  9]' 

2.94 

3 

-2.0 

10.35 

[2  10]' 

3.59 

4 

-2.0 

13.00 

[2  10]' 

4.25 

5 

-2.0 

15.60 

[2  10]' 

4.90 

6 

-2.0 

18.20 

[2  10]' 

5.55 

7 

-2.0 

20.80 

[2  10]' 

6.20 

8 

-2.0 

23.40 

[2  10]' 

6.85 

9 

-2.0 

26.00 

[2  10]' 

7.50 

10 

-2.0 

28.60 

[2  10]' 

8.15 

11 

-2.0 

31.20 

[2  10]' 

8.80 

12 

-2.0 

33.80 

[2  10]' 

9.45 

13 

-2.0 

36.40 

[2  10]' 

10.10 

The  surrogate  search  results  are  presented  graphically  in  Figures  7.4,  7.5,  and  7.6.  The  results 
indicate  that  cargo  up-load  rapidly  increases  for  the  first  few  steps  along  the  steepest  ascent  path, 


and  begins  to  slow  down  around  step  5  or  6.  Throughput,  on  the  other  hand,  increases  initially, 


with  the  increase  in  arrival  rate,  but  then  levels  off  at  step  3  and  4.  The  probability  that  an  arriving 
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aircraft  diverts  exhibits  a  steadily  increasing  value  as  the  analytical  model  is  evaluated  along  the 
steepest  ascent  path. 
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Figure  7.4  Surrogate  search  for  cargo  up-load  (tons/24  hours). 

As  part  of  the  study,  AMC  planners  do  not  want  the  probability  of  diverting  to  exceed  0.05. 
Prom  Figure  7.5  it  appears  that  this  threshold  is  crossed  somewhere  between  steps  1  and  2.  Here 
the  resolution  of  the  aircraft  proportion  within  the  analytical  model  did  not  allow  for  step  sizes 
that  could  explore  that  region.  So  the  aircraft  arrival  rate  that  results  in  a  divert  probability 
of  0.05  is  not  discernable  from  the  surrogate  search.  At  this  point,  a  modified  surrogate  search 
is  performed.  Since  the  analytical  model  can  be  evaluated  rapidly,  performing  more  than  one 
surrogate  search  is  stiQ  cost  efficient.  Note  that  the  time  to  complete  one  search  at  20  diSerent 
arrival  rates  is  approximately  15  seconds  on  a  266  MHZ  Pentium  II  PC  versus  approximately  10 
minutes  to  perform  20  replications  of  Psuedo-BRACE  at  one  design  point. 

Three  new  searches  are  conducted  to  locate  the  appropriate  arrival  rate  by  holding  aircraft 
proportions  constant  while  the  arrival  rate  is  varied.  For  each  of  the  three  searches,  the  aircraft 
proportion  is  held  constant  at  xi  =  1/3, /1/4,  and  1/6  respectively.  The  aircraft  arrival  rate  is 


_j - p. 


“T - r- 


+  + 


+  +  + 
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Figure  7.5  Surrogate  search  for  P(Divert). 

then  varied  in  increments  of  0.05  aircraft  per  hour  starting  at  1.30  aircraft  per  hour.  The  results  of 
the  searches  for  the  probability  of  diverting  and  cargo  up-load  are  presented  graphically  in  Figures 
7.7  and  7.8  respectively.  The  search  results  indicate  that  the  0.05  threshold  for  divert  probability 
is  crossed  when  the  aircraft  arrival  rate  is  close  1.7  to  1.8  aircraft  per  hour  depending  on  the 
aircraft  mix.  The  results  also  indicate  that  for  a  given  arrival  rate,  the  maximum  cargo  up-loaded 
is  provided  when  more  C-B  aircraft  are  used.  Hence  the  speed  of  the  analytical  model  is  exploited 
to  perform  several  surrogate  searches  in  order  to  make  up  for  a  lack  of  resolution  in  the  aircraft 
proportion  treatment  variable. 

We  now  must  validate  the  surrogate  search  results  using  Psuedo-BRACE.  We  will  focus  on 
the  probability  of  diverting,  since  the  problem  statement  constrains  the  system  to  operate  with  a 
probability  of  diverting  to  be  less  than  0.05.  We  choose  to  examine  the  results  of  the  surrogate  search 
when  the  proportion  of  C-A  aircraft  is  1/4.  This  translates  to  an  analytical  model  setting  of  three 
C-A  aircraft  and  9  C-B  aircraft.  When  the  aircraft  arrival  rate  is  set  to  1.75  aircraft  per  hour,  the 
analytical  model  estimates  P(Divert)  =  0.0502.  Validation  of  this  result  is  examined  by  generating 
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Figure  7.6  Surrogate  search  for  throughtput  (aircraft/hour). 

20  replications  of  Pseudo-BRACE  at  the  same  design  point.  The  estimated  values  for  each  of  the 
three  responses  are  compared  to  the  results  of  the  surrogate  search  at  this  design  point  in  Table 
7.13.  The  surrogate  model  produces  responses  for  the  mean  cargo  up-load  and  throughput  that  are 
reasonably  close  to  the  simulation  model.  However,  the  results  for  P(Divert)  are  unsatisfactory  for 
prediction  purposes.  This  is  not  totally  unexpected  since  each  model  computes  diverts  differently. 
We  expect  more  aircraft  to  divert  in  the  analytical  model  since  it  assumes  all  aircraft  arriving  to  a 
saturated  airfield  will  immediately  divert  whUe  the  simulation  model  allows  for  a  loiter  time  of  2 
hours. 


Table  7.13  Surrogate  search  verification  for  proportion  of  C-A  aircraft  =  1/4  and  aircraft  arrival 
rate  =  1.75. 


Response 

z 

Validation  Point  1. 

Cargo 

[1247.6  1272.0] 

P(Divert) 

[0.009  0.014] 

■IIIMM 

Throughput 

1.70 

1.66 
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Figure  7.7  Secondary  surrogate  searches  for  P(Divert). 

At  this  point,  the  analyst  has  essentially  three  options.  The  first  is  to  use  the  current  results 
to  find  an  appropriate  design  point  for  further  testing  using  the  simulation  model.  A  second  option 
is  to  return  to  the  conceptual  anal3rtical  model  development  phase  and  modify  the  model  so  that 
it  more  closely  approximates  Psuedo-BRACE.  The  third,  and  recommended  option  is  to  adjust 
the  output  of  the  anal3dical  model  based  on  the  results  of  the  simulation  replications  at  the  tested 
design  point.  In  a  sense,  we  are  creating  a  ‘‘new”  analytical  model  using  the  results  of  the  ciurent 
model. 

To  adjust  the  analytical  model  output  for  P(Divert)  to  more  closely  approximate  the  simu¬ 
lation  output,  we  first  assume  that  the  shape  of  the  response  surface  is  approximately  the  same 
for  both  models.  Thus  the  difference  recorded  at  the  first  validation  point  represents  a  constant 
difference,  or  translation,  between  the  two  surfaces.  Using  these  assumptions,  the  first  step  is  to 
approximate  the  response  surface,  or  curve  of  the  analytical  model  for  the  probability  of  divert¬ 
ing.  We  use  the  surrogate  search  results  and  the  simulation  response  at  validation  point  one  to 
approximate  the  P(Divert)  response  curve  for  proportion  of  C-A  aircraft  equals  1/4.  By  visually 
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Figure  7.8  Secondary  surrogate  search  for  cargo  (toiis/24  hours). 


inspecting  the  response  curve,  we  decide  to  approximate  it  using  a  second  order  curve  given  by 


P(Divert)^"  =  A  +  A^ 


where  A  is  the  aircraft  arrival  rate.  Using  least  squares  approximation  results  in 


F(Divert)^^  =  0.0584  -  0.1337A  +  0.0738A2 


(7.15) 


To  translate  the  approximated  response  curve,  we  set  A  =  1.75  and  P(Divert)®®  =  0.011  based  on 
the  Pseudo-BRACE  results.  Holding  and  constant,  we  solve  for  a  new  bg®  constant.  The 
final  approximation  is  then 


P(Divert)^^  =  0.0190  ~  0.1337A  +  0.0738A^ 


(7.16) 
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Using  the  surrogate  search  based  approximation,  we  solve  the  quadratic  equation 


0.05  =  0.0190  -  0.1337A  +  0.0738A^ 


(7.17) 


which  has  a  solution  of  approximately  2.03  aircraft  per  horn.  Based  on  this  value,  we  generate  20 
replications  of  Psuedo-BRACE  with  an  aircraft  arrival  rate  of  2.05  aircraft  per  hour.  The  results 
for  each  of  the  performance  measures  are  in  Table  7.14.  The  adjusted  analytical  model  provides 
a  much  more  accurate  approximation  of  P(Divert).  In  fact,  we  have  identified  a  point  within  the 
design  space  where  the  simulation  model  response  where  P (Divert)  =  0.050  is  contained  within  the 
90  percent  confidence  interval.  If  desired,  a  new  experimental  design  center  could  be  placed  at  this 
design  point  for  further  study  of  the  response  surface. 


Table  7.14  Surrogate  search  verification  for  proportion  of  C-A  aircraft  =  1/4  and  aircraft  arrival 
rate  =  2.05. 


90%  C.I. 

Z  Adjusted  Z 

Validation  Point  2. 

■Kl 

1380.4 

- 

P(Divert) 

■imr« 

0.104 

0.055 

Throughput 

1.94 

[1.92  1.96] 

lEE^I 

— 

In  summary,  we  have  shown  the  effective  application  of  the  surrogate  search  method  to  a 
simple  RSM  study.  The  application  demonstrates  several  characteristics  of  the  surrogate  search 
method.  First,  we  show  how  the  method  is  fiiUy  integrated  within  the  context  of  an  RSM  study. 
Once  a  conceptual  model  is  developed  and  translated  to  a  computer  program,  the  additional  steps 
for  operational  validity  are  performed  as  each  step  of  the  study  is  performed.  The  flexibility  of  the 
surrogate  search  methodology  has  also  been  demonstrated.  The  simple  study  provided  a  situation 
where  an  inadequate  initial  surrogate  search  can  be  rapidly  augmented  with  additional  modified 
searches.  In  this  case  the  modified  searches  were  made  with  one  of  the  treatments  held  constant. 
Finally,  we  demonstrate  a  simple  means  of  adjusting  the  output  of  the  anal3d;ical  model  when  it  is 
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believed  that  the  two  models’  response  surfaces  differ  by  only  a  constant.  The  method  provides  a 
rapid  means  of  refining  the  surrogate  search  model  output.  In  the  next  section,  we  demonstrate  the 
surrogate  search  method  on  an  actual  Air  Force  simulation  model  of  realistic  size  and  complexity. 

1.3  Airlift  Flow  Model  RSM  Study 

We  now  present  an  application  of  the  surrogate  search  method  using  the  USAF  Air  Mobility 
Command  (AMC)  Airlift  Flow  Model  (AFM)  simulation  model.  We  begin  with  a  brief  description 
of  the  airlift  system  and  the  AFM  simulation  model.  Next,  we  discuss  an  AFM  scenario  that  has 
been  developed  for  academic  research  purposes.  We  then  present  an  RSM  study  based  on  the 
academic  scenario  that  will  be  examined  using  the  surrogate  search  methodology.  This  is  followed 
by  a  description  of  the  particular  settings  of  the  AFM  model  in  order  to  complete  the  study. 
We  then  present  the  analytical  model  used  as  a  surrogate  and  the  results  of  the  surrogate  search 
validation  process.  We  conclude  with  the  results  of  the  study  using  the  surrogate  search  method. 

7.3.1  The  Airlift  System  and  AFM.  AMC  is  responsible  for  providing  global  airlift  of 
cargo  and  troops  in  support  of  the  Department  of  Defense  (DOD).  The  airlift  system  consists  mainly 
of  military  aircraft,  aircrews,  airfields,  air  routes,  air  refueling,  support  equipment  and  personnel, 
fuel,  and  the  airlift  movement  requirements.  Under  certain  conditions,  the  National  Command 
Authority  can  also  task  civilian  airlines  to  provide  aircraft  in  support  of  the  airlift  mission.  These 
aircraft  are  referred  to  as  the  Civil  Reserve  Air  Fleet  (CRAF)  and  when  activated  fall  under  the 
authority  of  AMC.  The  airlift  movement  requirements  for  any  particular  DOD  tasking,  or  plan, 
are  contained  within  a  Time-Phased  Force  Deployment  Data  (TPFDD)  document.  The  TPFDD 
includes  the  on-load  location,  on-load  availability  day,  required  off-load  location,  and  required 
delivery  day  for  each  requirement.  It  is  then  up  to  the  AMC  planners  to  plan  the  necessary  airlift 
missions  to  meet  the  TPFDD  requirements. 
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AMC  Tises  the  Mobility  Analysis  Support  System  (MASS)  simulation  model  as  an  analysis 
tool  to  support  decision  making  related  to  the  airlift  system.  The  core  of  the  model  is  the  Airlift 
Flow  Model  (AFM),  which  simulates  the  global  airlift  system.  AFM  is  a  stochastic  discrete  event 
simulation  model.  It  is  capable  of  simulating  AMC  policies,  procedures,  operations,  aircraft,  air 
bases,  cargo,  passengers,  and  support  resources  [11].  AFM  simulates  a  fleet  of  aircraft  moving  a 
given  amount  of  cargo  and  passengers  from  an  on-load  point,  through  any  needed  en-route  stops, 
to  an  off-load  point,  then  recovering  and  returning  to  home  station  for  another  mission.  The  model 
can  continue  this  process  for  as  many  simulated  days  as  desired,  or  until  all  requirements  have  been 
airlifted  to  their  destination  [11].  This  process  is  referred  to  as  executing  a  scenario.  To  accomplish 
a  scenario,  AFM  performs  three  major  tasks:  simulation  control,  mission  planning,  and  mission 
execution.  The  relationship  between  the  three  functions  is  depicted  in  Figure  7.9. 


Figure  7.9  Airlift  Flow  Model  (AFM)  functionality  relationship. 


The  following  excerpt  from  an  AMC  Studies  and  Analysis  Flight  point  paper  [33]  provides 
an  excellent  synopsis  of  AFM: 
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AFM  Inputs  Possible  AFM  inputs  include: 

•  A  TPFDD  document  containing  airlift  movement  requirements. 

•  An  airlift  network  consisting  of  on-loads,  off-loads,  en-route  stops,  recovery  bases,  and 
home  stations  connected  by  user-defined  routes. 

•  An  airlift  fleet  of  different  aircraft  types  identified  by  individual  tail  numbers. 

•  Individual  aircrews  who  must  be  available  to  allow  missions  to  be  flown. 

•  Logistic  factors  which  account  for  refueling,  maintenance,  and  material  handling  of  cargo. 

•  Concepts  of  operations  that  include  strategic  inter-theater  airlift,  aerial  refueling,  intra¬ 
theater  shuttle  operations,  direct  delivery  operations,  and  recovery/stage  operations. 

Mission  Planning  AFM  performs  the  following  mission  planning  tasks: 

•  Prioritization  of  requirements  by  available-to-load  dates  and  required  delivery  dates. 

•  Prioritized  route  selection  and  reservation  for  flight  planning. 

•  Marrying  a  specific  aircraft  tail  number  to  the  next  eligible  requirement. 

•  Crew  planning  to  ensure  that  only  the  crews  eligible  to  fly  do  fly. 

Mission  Execution  AFM  simulates  mission  execution  by  simulating: 

•  Typical  sortie  events  including:  taxi-out,  takeoff,  departure,  in-route  cruise,  initial  ap¬ 
proach,  final  approach,  landing,  taxi-in,  and  ground  activities  for  every  sortie  of  the 
mission. 

•  Ground  activity  resource  allocation  and  planned  delays  for:  ramp  space,  off-loading  and 
on-loading  cargo,  refueling,  maintenance,  and  crew  changing. 

•  Optionally,  detailed  loading  of  each  piece  of  cargo  for  compatibility  with  the  doors  and 
remaining  space  on  each  aircraft. 
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•  Crew  activities  and  monitoring  events,  including:  crew  rest,  crew  monthly  and  quarterly 
flying  hour  limits,  crew  availability,  and  searches  for  unavailable  crews. 

AFM  Outputs  AFM  output  includes: 

•  Aircraft  related  statistics  such  as:  utilization  rates,  payload,  ground  service  time,  flight 
time,  and  system  delays. 

•  Aircrew  related  statistics  such  as:  crew  duty  days,  number  of  crews,  hours  flown  by  each 
crew,  and  crew  availability. 

•  Cargo  related  statistics:  total  tons  delivered,  tons  per  day  throughput,  unit  and  force 
closure,  actual  million  tons  miles  per  day  flown,  and  cargo  remaining  in  backlog. 

•  Airlift  network  statistics:  typical  cycle  times,  flying  times,  network  airfield  use,  maximum 
on  ground  (MOG)  constraints,  and  system  bottlenecks. 

7,3,2  AFM  Academic  Scenario.  We  begin  by  examining  a  notional  AFM  strategic  inter¬ 
theater  airlift  scenario.  Since  almost  all  scenarios  studied  by  AMC  contain  classified  information, 
an  academic  scenario  of  realistic  size  and  complexity  has  been  developed.  The  scenario’s  nominal 
aircraft  fleet  consists  of  185  total  aircraft,  including  C-5’s,  C-17’s,  C-141’s,  and  GRAF  aircraft. 
The  TPFDD  contains  an  airlift  requirement  of  approximately  26,000  tons  of  cargo  and  35,700 
passengers,  which  become  available  for  movement  over  a  period  of  15  days.  The  primary  on¬ 
load  locations  are  McGuire  and  Charleston  Air  Force  Bases  (AFB’s)  and  John  F.  Kennedy  (JFK) 
International  Airport.  The  primary  off-load  locations  are  in  Bahrain,  Dhahran,  and  King  Abdul 
Aziz  Intemational  Airports  in  Saudi  Arabia.  AU  requirements  are  to  be  delivered  within  21  days. 
The  airlift  missions  beginning  in  the  CONUS  are  either  flown  directly  to  their  destination  or  are 
routed  through  one  of  the  several  en-route  air  bases  located  in  Europe.  The  en-route  air  bases  for 
the  military  aircraft  are  Mildenhall  Air  Base  (AB),  England;  Ramstein  AB,  Germany;  Moron  AB, 
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Spain;  and  Rota  Naval  Air  Station,  Spain.  The  CRAF  aircraft  use  Heathrow  International  Airport, 
T^ngland  and  Frankfurt  Main  International  Airport,  Germany  as  their  en-route  bases. 

The  nominal  aircraft  fleet  consists  of  60  C-5’s,  50  C-17’s,  50  C-141’s,  and  25  CRAF  aircraft. 
Half  of  these  aircraft  become  available  for  missions  on  the  first  day  of  the  scenario  with  the  other 
half  available  on  the  second  day.  The  home  base  for  half  of  the  C-141’s  and  C-5’s  is  McGuire 
AFB  with  the  remaining  aircraft’s  home  base  is  Charleston  AFB.  All  of  the  C-17’s  are  based  at 
Charleston  AFB  and  all  of  the  CRAF  aircraft  have  JFK  as  their  home  station. 

In  the  real  airlift  system,  the  availabihty  of  different  resources  at  each  base  determines  the 
rate  at  which  aircraft  are  serviced.  These  resources  include  parking  spaces,  fuel,  parts,  support 
equipment,  and  different  support  personnel  and  AFM  has  the  ability  to  simulate  these  different 
resources.  For  this  scenario  these  resources,  other  than  fuel,  are  aggregated  within  the  single 
resource  called  maximum  on  the  ground  (MOG).  For  each  air  base  in  the  scenario,  a  sin^e  value  of 
MOG  is  input  representing  the  maximum  nTimber  of  aircraft  that  an  air  base  can  support  at  one 
time.  Any  aircraft  that  arrives  at  a  base  such  that  the  number  of  aircraft  on  the  base  now  exceeds 
the  MOG  value  for  that  base  must  wait  for  a  MOG  resource  to  be  freed  before  any  servicing  is 
begun  on  that  aircraft.  All  servicing  is  then  based  on  a  constant  time  for  the  type  of  servicing  being 
performed  (off-load,  en-route,  on-load,  etc.  )  and  the  type  of  aircraft.  Based  on  this,  the  mission 
planniTig  fimptinn  of  AFM  must  reserve  MOG  at  the  air  bases  necessary  to  complete  a  particular 
mission  before  a  mission  can  be  planned  [11].  In  this  scenario,  only  McGuire  AFB,  Charleston 
AFB,  and  the  civilian  airports  have  unlimited  MOG. 

For  every  hour  an  aircraft  flies,  a  certain  amount  of  ground  servicing  and  repair  time  is 
required  before  the  aircraft  is  capable  of  flying  again.  The  long  nm  average  of  time  a  particular 
t3rpe  of  aircraft  is  capable  of  flying  in  a  24  hotu  period  of  time  is  called  the  aircraft’s  capability  rate. 
Several  factors  determine  this  capability  rate  for  each  type  of  aircraft.  The  factors  include  aircraft 
system  reliabihty,  scheduled  maintenance  requirements,  spare  part  availability,  and  average  ground 
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servicing  times.  Since  AFM  uses  a  constant  ground  time  for  groimd  servicing  in  this  scenario, 
some  other  means  of  simulating  this  capability  rate  is  required.  These  aircraft  capability  rates  are 
simulated  in  AFM  in  the  academic  scenario  using  two  procedures,  the  capping  procedure  and  the 
differential  procedure. 

The  AFM  capping  procedure  relies  upon  a  user  input  of  aircraft  capability  rates  based  on 
historical  data.  Based  on  the  munber  of  aircraft  in  the  scenario,  the  capping  procedme  determines 
the  number  of  available  flying  hours  for  each  day.  It  then  monitors  the  actual  flying  hours  flown, 
and  planned,  during  AFM  execution.  Once  the  flying  hour  cap  is  reached,  the  capping  procedure 
prevents  the  planning  of  any  more  missions  until  more  flying  hours  become  available.  This  procedure 
can  work  well,  however  there  are  some  situations  where  the  application  of  the  procedure  results  in 
imreaJistic  aircraft  activity  [11]. 

The  differential  procedure  was  developed  to  address  the  problems  encountered  in  the  capping 
procedure  [11].  This  procedure  randomly  groimds  aircraft  at  their  home  station  so  that  the  amount 
of  flying  accomplished  tends  toward  the  input  capability  rate.  The  differential  procedure  accom¬ 
plishes  this  ly  tracking  the  history  of  all  home  station  visits  realized  during  an  AFM  replication. 
Based  on  the  observed  history,  aircraft  are  periodically  grounded  at  their  home  station  based  on  the 
solution  to  a  transportation  control  problem  that  determines  the  amount  of  additional  grotmd  time 
required  to  meet  the  desired  capability  rate  [llj.  Both  procedures  are  then  applied  to  realistically 
simulate  the  historical  aircraft  capability  rates. 

1.3.3  RSM  Problem  Statement.  Of  the  many  possible  performance  measures  for  this 
scenario,  we  focus  on  the  delivery  of  cargo  and  passengers.  We  can  measure  cargo  and  passengers 
with  a  single  value  by  considering  passengers  and  their  gear  to  weigh  350  poimds  each.  Therefore, 
for  the  rest  of  this  section  we  consider  all  requirements  airlifted  as  tons  of  cargo.  An  obvious  problem 
statement  that  we  first  considered  is  that  of  finding  the  optimal  mix  of  aircraft  that  maximizes  the 
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tons  of  cargo  delivered  per  day.  However,  there  are  some  inherent  problems  with  that  type  of 
problem  statement  in  the  context  of  this  scenario. 

For  the  academic  scenario,  and  any  real-world  scenario  of  this  type,  there  are  two  factors, 
other  than  the  amount  of  aircraft,  which  limit  the  amount  of  cargo  that  can  be  delivered  each  day. 
The  iBrst  factor  is  the  TPFDD  requirement.  Obviously,  only  a  finite  amount  of  cargo  is  available 
for  movement  on  any  given  day  as  defined  by  the  TPFDD  document.  The  second  factor  is  the 
amoimt  of  MOG  available  throughout  the  airlift  network.  In  other  words,  there  is  a  finite  limit  to 
the  number  of  aircraft  that  can  be  serviced  and/or  off-loaded  at  any  given  time  at  the  en-route 
and  delivery  destination  air  bases.  Therefore,  as  the  number  of  aircraft  in  the  scenario  is  increased, 
one  of  these  two  factors  will  act  as  a  limit  to  the  amoimt  of  cargo  that  can  be  moved  per  day.  If 
the  network  can  handle  it,  that  limit  will  be  defined  by  the  TPFDD,  Otherwise,  the  constrained 
capacity  of  the  airlift  network  will  limit  the  amount  of  cargo  delivered  per  day  at  a  level  less  than 
that  made  available  by  the  TPFDD. 

While  the  maximization  of  cargo  delivery  rates  is  important,  it  is  not  necessarily  the  goal  that 
AMC  strives  to  meet  during  a  particular  airlift  tasking.  Actually,  the  primary  goal  is  to  deliver 
the  cargo  where  it  is  required,  when  it  is  required,  as  defined  by  the  TPFDD.  In  order  to  meet 
the  academic  scenario’s  delivery  requirements,  it  is  not  necessary  to  deliver  the  cargo  at  the  same 
rate  that  it  is  made  available.  This  is  clear  since  all  cargo  is  available  for  movement  in  15  days 
while  it’s  delivery  requirement  occurs  typically  5  days  later.  Therefore,  we  can  redefine  the  limiting 
factors  in  terms  of  meeting  the  TPFDD  delivery  requirements.  During  our  initial  investigations 
of  the  academic  scenario  we  discovered  that  the  network  can  service  enough  aircraft  so  that  the 
TPFDD  delivery  requirement  can  be  met.  Further,  we  discovered  that  the  nominal  aircraft  fleet  of 
185  aircraft  contains  excess  capacity.  By  that  we  mean  the  TPFDD  delivery  requirement  can  be 
met  with  less  than  185  aircraft. 


7-30 


Based  on  the  above  discussion,  we  wish  to  find  the  optimal  mix  of  aircraft  that  most  “ef¬ 
ficiently”  delivers  the  TPFDD  requirement  on  time.  The  task  then  is  to  define  an  appropriate 
measure  of  effectiveness  that  rewards  on-time  or  early  delivery  of  cargo  and  penalizes  excess  airlift 
capacity.  Hence  the  measure  should  be  a  function  of  the  number  of  aircraft,  the  amount  of  cargo 
delivered,  the  amount  of  time  that  it  takes  to  deliver  the  cargo,  and  the  TPFDD  required  delivery 
time.  Based  on  this  we  develop  two  measures  of  effectiveness  described  below. 

1£  we  wish  to  maximize  our  efficient  delivery  of  cargo,  we  can  translate  that  to  a  desire  to 
maximize  the  amount  of  on-time,  or  early,  cargo  each  aircraft  delivers.  To  do  this  we  measure  the 
average  difference  between  the  accumulated  delivered  cargo  and  the  required  accumulated  cargo 
per  day,  per  aircraft.  We  call  this  the  early  cargo  per  aircraft  (EGA)  measure  of  effectiveness.  If  we 
are  at,  or  near,  the  TnayimuTn  throughput  of  the  network/TPFDD  constraints,  adding  more  aircraft 
wiU  decrease  the  value  of  EGA.  Gonversely  removing  aircraft  will  increase  the  value  of  EGA  imtil 
the  cargo  throughput  drops  below  some  threshold.  Thus,  EGA  provides  a  measure  of  the  efficient 
effectiveness  of  the  aircraft  fieet. 

Another  approach  is  to  multiply  the  number  of  aircraft  in  the  fleet  by  a  measure  of  their 
effective  efficiency.  Thus,  our  goal  is  to  minimize  this  measure,  which  we  call  the  aircraft-cargo 
ratio  (AGR).  To  measure  the  effective  efficiency  we  consider  two  ratios.  The  first  ratio  compares 
the  amount  of  time  realized  to  close  out  a  TPFDD  versus  the  required  close  out  time.  If  the  TPFDD 
is  closed  out  in  less  time  than  required,  this  ratio  is  less  than  one,  reducing  the  value  of  the  AGR. 
Taking  longer  than  planned  causes  the  ratio  to  be  greater  than  one  and  AGR  grows.  The  second 
ratio  compares  the  amoimt  of  cargo  in  the  TPFDD  to  the  amoimt  that  is  actually  delivered  by  the 
last  required  delivery  date.  If  aU  the  cargo  is  moved  by  the  last  day  of  the  TPFFD,  this  ratio  will 
equal  one.  Otherwise,  if  some  of  the  cargo  is  delivered  after  the  last  required  delivery  date,  this 
ratio  will  be  greater  than  one  causing  an  increase  in  the  AGR.  We  must  make  an  adjustment  to 
this  measure  to  account  for  situations  where  the  total  number  of  aircraft  in  the  fleet  is  particularly 
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low,  causing  a  “false”  low  value  of  ACR.  When  the  amount  of  time  required  to  close  out  the 
TPFDD  exceeds  the  required  time  we  add  additional  “aircraft”  to  the  value  of  ACR,  The  number 
of  additional  aircraft  added  is  equal  to  the  amount  of  cargo  undelivered  by  the  TPFDD  close  out 
date,  divided  by  the  average  aircraft  cargo  load  observed.  In  this  manner,  the  effective  efficiency 
of  the  aircraft  fleet  is  measured  by  the  ACR. 

We  now  derive  AFM  simulation  output  statistics  to  estimate  ECA  and  ACR.  The  method 
of  independent  replications  is  used  to  estimate  each  of  the  measures  and  the  notation  is  developed 
with  that  method  in  mind.  We  let  the  number  of  replications  generated  at  a  design  point  equal  n. 
Since  the  two  measures  share  many  of  the  same  terms,  we  develop  the  following  shared  notation. 
First  let  the  number  of  aircraft  in  the  aircraft  fleet  used  in  a  scenario,  or  experimental  design  point, 
be  denoted  by  A.  We  now  address  the  TPFDD  cargo  requirements,  which  are  constant  throughout 
our  study.  Each  line  of  the  TPFDD  denotes  a  consolidated  “package”  of  cargo  and/or  passengers 
available  at  a  single  airbase  that  is  required  at  a  single  airbase  in  the  theater  of  operations.  The 
package  is  assigned  a  date  (number  of  days  from  the  beginning  of  the  scenario)  when  it  is  available 
for  loading  on  an  aircraft,  and  a  date  when  it  is  required  for  delivery.  We  track  the  amount  of 
cargo  that  is  required  for  delivery  each  day  by  first  letting  i  represent  the  scenario  day  such  that 
1  <  i  <  21.  Then  let  CR{i)  be  the  amount  of  cargo,  in  tons,  that  the  TPFDD  designates  for  delivery 
on  day  i  over  the  entire  airlift  system.  Thus  the  accumulated  TPFDD  cargo  requirement  for  the 
day  of  the  scenario,  Cj2(fc),  is  given  by 


=  (7.18) 

i<k 

In  a  similar  manner,  we  represent  the  cargo  that  is  actually  delivered  on  day  i  during  replication  j 
as  Thus,  the  accumulated  cargo  delivered  by  day  fc  in  the  scenario  for  replication  j,  C^{k)j, 
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is  computed  as 


CS Wi  =  fe  =  1, 2, . . .  ,  21  i  =  1, 2, . . .  ,  n  (7.19) 

We  now  represent  the  number  of  days  required  to  deliver  the  entire  TPFDD  cargo  requirement  for 
replication  j  as  which  is  deiBned  mathematically  as 


TBj  =  k  3  {k  =  l,2,..-\  =  CRi21)  and  C^k  -  l)j  <  ^^(21)}  (7.20) 

We  also  let  represent  the  number  of  hours  required  to  deliver  all  cargo  for  replication  j  such 
that  24Tg.  -  24  <  <  24T|^.. 

We  estimate  early  cargo  per  aircraft,  EC  A,  using  AFM  in  the  following  manner.  For  each 
replication  j  we  calculate  the  realized  ECAj  by 

EC  A!  =  ^  E  ,  i  =  1,2, . . .  ,n  (7.21) 

i=l 

Thus  the  estimate  of  EGA  using  the  method  of  independent  replications  is  given  by 

ECA^  =  n-^ 

j=i 


The  aircraft-cargo  ratio,  ACR  is  estimated  by  first  considering  the  effective  efficiency  ratios. 
First  consider  the  ratio  of  the  time  required  to  deliver  the  TPFDD  requirement  during  replication 
j  to  the  amount  of  time  before  the  TPFDD  requires  all  cargo  to  be  delivered,  Dj  ,  found  by 


7  “  1  2  TL 

”  24-21’  ^  ’  ’■■■  ’ 


(7.23) 
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Now  consider  the  ratio  of  the  total  cargo  requirement  to  the  accumulated  cargo  delivered  during 
replication  j  given  by 


Cr{21) 

Cl,i2lV 


j  — 1)2,...  ,n 


(7.24) 


If  we  let  Gj  be  the  average  cargo  load  of  all  cargo  deliveries  in  replication  j,  the  estimated  ACR 
for  replication  j  is  calculated  as 


ACRf  =  ■{ 


A{Df){M?) 


ifT«^<21 
if  T^j  >  21 


(7.25) 


SO  that  ACR  can  be  estimated  by 


ACR^  =  n-^Yl 

j=i 


We  perform  an  RSM  study  using  both  measures  of  effectiveness,  where  we  wish  to  maximize 
EGA  and  minimize  ACR.  We  don’t  know  if  the  optimal  value  for  each  of  these  two  measures  occurs 
at  the  same  design  point,  so  our  goal  is  to  construct  a  response  surface  that  contains  the  optimal 
value  for  each  measure.  The  design  factors,  or  treatments,  for  the  study  are  the  number  of  each  of 
the  four  types  of  aircraft  in  the  aircraft  fleet.  We  assign  the  factors  by 


xi  =  number  of  CRAF  aircraft 
Xo  =  number  of  C  —  141  aircraft 

(7.27) 

x^  =  number  of  C  5  aircraft 
X4  =  number  of  C  —  17  aircraft 

To  construct  an  experimental  design  to  estimate  the  first  order  response  surface  for  each  measure 
we  construct  a  2^  factorial  design  in  the  following  manner.  Using  the  academic  scenario  nominal 
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settings,  we  assign  the  design  center  as  xi  =  25,  X2  =  50,  xz  =  60,  and  x^  =  50,  where  we  vary  xi 
by  plus  and  minus  5  aircraft,  and  the  other  three  treatments  by  plus  and  minus  10  aircraft  each. 
The  levels  of  the  coded  treatment  variables,  0  =  (©i,  02,  ©3.  ©4)'  are  found  using  the  following 
formulas 


©i  = 

-25 

5  ’ 

j  =  H,L 

(7.28) 

©2  “ 

a4  —  50 
10  ’ 

j  =  H,L 

(7.29) 

©3  = 

a4  —  60 
10  ’ 

j  =  H,L 

(7.30) 

©i  = 

a4  —  50 
10  ’ 

j  =  H,L 

(7.31) 

The  resulting  coded  and  uncoded  experimental  design  appears  in  Table  7.15. 


Table  7.15  Initial  AFM  RSM  2*  fectorial  design. 


Design 

Coded 

Treatment  Levels 

Uncoded 
Treatment  Levels 

Point 

©1 

©2 

©3 

©4 

Xi 

X2 

^3 

1 

1 

1 

1 

1 

30 

60 

70 

60 

2 

1 

1 

1 

-1 

30 

60 

70 

40 

3 

1 

1 

-1 

1 

30 

60 

50 

60 

4 

1 

1 

-1 

-1 

30 

60 

50 

40 

5 

1 

-1 

1 

1 

30 

40 

70 

60 

6 

1 

-1 

1 

-1 

30 

40 

70 

40 

7 

1 

-1 

-1 

1 

30 

40 

50 

60 

8 

1 

-1 

-1 

-1 

30 

40 

50 

40 

9 

-1 

1 

1 

1 

20 

60 

70 

60 

10 

-1 

1 

1 

-1 

20 

60 

70 

40 

11 

-1 

1 

-1 

1 

20 

60 

50 

60 

12 

-1 

1 

-1 

-1 

20 

60 

50 

40 

13 

-1 

-1 

1 

1 

20 

40 

70 

60 

14 

-1 

-1 

1 

-1 

20 

40 

70 

40 

15 

-1 

-1 

-1 

1 

20 

40 

50 

60 

16 

-1 

-1 

-1 

-1 

20 

40 

50 

40 

7.3.4  AFM  Settings.  We  describe  the  specific  aircraft  and  airfield  parameters  addressed 
above  in  section  7.3.2.  We  begin  by  listing  the  aircraft  parameters  in  Table  7.16.  The  parameters 
include  type  of  body  (wide  or  narrow),  capability  rate,  and  the  standard  times  associated  with 
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oflF-load  servicing,  on-load  servicing,  and  en-route  servicing.  The  body  type  determines  the  amoimt 
of  ramp  space  and  associated  MOG  that  an  aircraft  occupies.  Essentially,  a  wide  body  aircraft 
occupies  twice  as  much  MOG  as  a  narrow  body  aircraft.  The  capability  rate  indicates  the  long 
range  average  operating  time  for  every  24  hours.  Recall  that  the  AFM  capping  and  differential 
procedures  use  the  capability  rate  input  to  simulate  aircraft  operations.  For  the  standard  service 
times,  the  en-route  time  is  also  used  as  the  standard  recovery  time  at  an  aircraft’s  recovery  or  home 
base.  The  MOG  capacity  for  each  air  base  in  the  scenario,  by  aircraft  body  type,  is  listed  in  Table 
7.17.  Each  of  these  settings  remain  constant  at  each  of  the  16  experimental  design  points. 


Ihble  7.16 

AFM  RSM  study  aircraft  parameters. 

Aircraft 

Type 

Body 

Type 

Capability 

Standard  Service  Times  (hrs) 

Rate  (hrs) 

En-Route 

Off-Load 

On-Load 

GRAF 

Wide 

12.0 

1.50 

2.00 

3.50 

C-141 

Narrow 

12.2 

2.25 

2.25 

2.25 

C-5 

Wide 

10.7 

3.25 

3.25 

4.25 

C-17 

Narrow 

15.3 

2.25 

2.25 

2.25 

Table  7.17  AFM  RSM  study  airbase  MOG  capacities. 

Air  Base 

MOG 

Narrow  Wide 

Charleston  AFB 

Unlimited 

Unlimited 

McGuire  AFB 

Unlimited 

Unlimited 

JFKIAP 

Unlimited 

Unlimited 

Ramstein  AB 

9 

4 

Mildenhall  AB 

4 

2 

Barajas  lAP 

28 

14 

Moron  AB 

4 

4 

Rota  NAS 

2 

1 

Frankfurt  Main  lAP 

Unlinuted 

Unlimited 

Heathrow  lAP 

32 

16 

Bahrain  LAP 

26 

13 

Dhahran  LAP 

10 

5 

King  Abdul  Aziz  LAP 

13 

6 

1.3.5  Surrogate  Search  Validation  and  Initial  RSM  Results.  The  surrogate  search  vali¬ 
dation  and  initial  RSM  results  are  presented  in  the  following  section.  We  begin  by  describing  and 


7-36 


validating  the  proposed  conceptual  analytical  model  of  the  AFM  simulation  model.  Next  we  estab¬ 
lish  surrogate  search  operational  validity  while  performing  and  analyzing  the  initial  experiments  of 
the  RSM  study. 

7,3,5 A  AFM  Conceptual  Analytical  Model  Validity.  The  next  step  in  the  surrogate 
search  methodology  is  to  propose  and  validate  a  conceptual  analytical  model  of  the  credible  sim¬ 
ulation  model  AFM.  As  presented  in  Section  6.3.1  we  validate  the  conceptual  analytical  model  by 
first  establishing  face  validity  and  then  ensure  that  the  analytical  model  possesses  the  appropriate 
output  performance  measures  and  treatment  parameter  inputs.  See  Figure  7.1  for  a  flow  chart  of 
the  described  process.  We  begin  by  first  describing  the  proposed  conceptual  model  of  AFM.  We 
then  examine  the  choices  made  in  creating  the  model  as  we  attempt  to  establish  face  validity  of  the 
model.  This  section  is  then  concluded  with  an  investigation  into  the  presence  of  the  appropriate 
output  and  input  parameters. 

We  propose  a  multi-class  closed  queueing  network  model  solved  via  the  MVA  algorithm  as 
the  conceptual  analytical  model  of  AFM.  The  customers,  or  entities,  within  the  network  are  the 
airlift  aircraft  simulated  in  AFM.  We  initially  consider  four  classes  of  customers  corresponding  to 
the  four  types  of  aircraft  in  the  AFM  scenario.  Thus,  for  the  RSM  design  center  we  have  a  total  of 
185  customers  spread  appropriately  across  the  four  classes.  We  begin  by  representing  each  of  the 
13  air  bases  as  a  single  service  center  with  an  infinite  number  of  servers.  The  service  time  at  each 
of  the  air  bases  is  established  by  observing  the  realized  mean  time  aircraft  spend  at  the  appropriate 
air  base  during  AFM  realizations.  The  air  routes  from  one  air  base  to  another  are  also  represented 
as  a  service  center  with  an  infinite  number  of  servers  (or  equivalently  as  a  delay  station  [8])  where 
the  mean  service  time  is  found  by  observing  realizations  of  the  AFM  sunulation  model.  For  the  13 
air  bases  there  are  possibly  (13  -  1)^^  =  12^^  =  156  possible  air  routes  for  each  class  of  aircraft 
resulting  m  a  total  of  12^^  +  13  =  169  possible  service  centers  for  each  aircraft  class.  Based  upon 
an  input  to  AFM  of  permissible  aircraft  routings,  and  initial  observations  of  AFM,  we  don’t  expect 
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the  conceptual  model  to  require  all  169  service  centers,  however  the  model  size  will  be  fairly  large 
none  the  less.  As  with  the  mean  service  times,  the  probabilities  that  define  the  four  probability 
transition  matrices  of  this  model  are  determined  hy  observing  AFM  simulation  realizations. 

A  primary  concern  when  creating  an  analytical  model  for  the  purposes  of  surrogate  searches 
and  generating  an  ACV  is  the  relative  speed  at  which  the  ACV  is  computed  when  compared  to 
the  simulation  model.  Initial  testing  of  this  analytical  model  at  design  point  one  with  215  aircraft 
(customers)  in  4  classes  resulted  in  computation  times  of  approximately  194  minutes.  Since  10 
replications  of  the  AFM  model  at  a  design  point  only  takes  approximately  15  minutes  to  generate, 
the  computation  time  of  the  4  class  analytical  model  is  unsatisfactory.  It  is  well  known  that  the 
number  of  numerical  operations  when  computing  the  MVA  algorithm  grows  exponentially  with  the 
number  of  customer  classes  [16].  Therefore,  we  choose  to  reduce  the  number  of  customer  classes 
from  4  to  2  by  grouping  all  of  the  military  aircraft  into  a  single  class.  This  is  a  natural  choice  since 
the  military  aircraft  share  the  same  home  stations  and  en-route  bases  and  the  CRAF  aircraft  have 
a  diflferent  home  station  and  set  of  en-route  bases.  Making  this  adjustment  reduces  computation 
time  at  the  same  design  point  from  194  minutes  to  8.6  seconds. 

We  now  consider  whether  the  proposed  conceptual  analytical  model  is  a  reasonable  model 
of  AFM  and  thus  meets  the  requirement  of  face  validity.  Given  that  we  can  correctly  identify  the 
model  parameters,  the  basic  assumption  that  AFM  can  be  modeled  as  a  closed  queueing  network 
certainly  seems  reasonable.  After  all,  in  AFM  we  have  aircraft  (customers)  flowing  through  a  closed 
airlift  system  consisting  of  13  air  bases  and  the  appropriate  set  of  air  routes  that  connect  them. 
However,  the  performance  measures  calculated  by  the  MVA  algorithm  are  based  on  the  assumption 
that  the  closed  queueing  network  has  reached  “steady  state”  operation  [29].  The  output  statistics 
of  the  AFM  model  are  the  result  of  a  terminating  simulation  model  that  doesn’t  necessarily  begin  to 
exhibit  steady  state  behavior.  Indeed,  the  output  statistics  include  data  from  the  beginning  of  each 
replication  when  AFM  is  certainly  not  operating  at  steady  state.  The  alternative  is  to  construct 
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an  analytical  model  that  doesn’t  assume  steady  state  behavior,  however  finding  a  solution  for 
non-steady  state  queues  is  dfficult  for  even  the  simplest  of  queueing  models  [28].  Even  if  such  a 
solution  could  be  found  for  AFM,  it  is  assumed  that  the  complexity  of  such  of  model  would  prohibit 
it’s  eflacient  use.  Thus,  constructing  an  analytical  model  that  approximates  AFM  using  the  MVA 
algorithm  approach  seems  to  be  the  best  alternative. 

We  now  investigate  the  reasonableness  of  the  settings  and  parameters  of  the  proposed  closed 
queueing  network.  It  certainly  seems  appropriate  that  the  air  routes  are  modeled  as  delay  stations 
with  service  times  equal  to  the  realized  transit  times  of  the  AFM  model.  What  is  not  obvious  is 
whether  or  not  the  air  bases  are  appropriately  modeled  as  delay  stations  and  if  the  decision  processes 
involved  in  creating  airlift  missions  can  be  adequately  modeled  by  a  transition  probability  matrix 
based  on  the  observed  routings  in  AFM.  Further,  even  if  these  modeling  decisions  are  appropriate, 
we  must  also  decide  if  the  grouping  of  all  military  aircraft  into  a  single  class  is  appropriate. 

When  an  aircraft  arrives  at  an  airbase  that  has  an  available  MOG  resource  to  either  on-load 
or  off-load  cargo,  or  for  en-route  servicing,  it  spends  a  fixed  amount  of  time  at  that  air  field  based 
on  the  standard  servicing  times  in  Table  7.16.  As  long  as  there  is  an  available  MOG  resource,  the 
aircraft  doesn’t  wait  (queue)  for  service  and  the  time  in  service  is  the  same  regardless  of  the  number 
of  aircraft  also  in  service.  Further,  if  the  modeler  carefully  selects  the  AFM  input  parameters  for 
allowable  aircraft  routings,  AFM  will  schedule  missions  and  reserve  MOG  so  that  each  air  base  will 
have  enough  MOG  available  for  all  arriving  aircraft.  Therefore,  it  certainly  seems  appropriate  to 
model  the  en-route  and  off-load  bases  as  delay  stations. 

Aircraft  behave  differently  at  their  home  stations.  Two  factors  interact  to  determine  the 
amoimt  of  time  an  aircraft  spends  at  home  station.  The  first  factor  is  the  combined  capping  and 
differential  procediue  that  groimd  aircraft  at  their  home  station  in  order  to  simulate  the  aircraft 
capability  rate.  The  other  factor  consists  of  waiting  in  a  first-in-first-out  queue  of  all  aircraft  at  each 
of  the  three  home  stations  for  an  airlift  mission  to  be  planned.  The  planning  of  a  mission  depends 
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on  the  availability  of  cargo  and  the  availability  of  sufficient  MOG  to  complete  the  mission  [11].  In 
this  case  it  not  clear  if  modeling  the  home  stations  as  delay  stations  is  appropriate.  However,  it  is 
also  not  clear  how  to  model  it  differently.  The  MVA  algorithm  cannot  handle  a  queue  of  customers 
from  more  than  one  service  station  [29].  So  at  this  point,  we  decide  to  use  the  delay  station  model 
at  all  air  bases  and  test  our  decision  during  surrogate  search  operational  validation. 

The  routing  of  aircraft  within  AFM  is  a  complex  process.  As  mentioned  above,  it  is  a  function 
of  available  aircraft  flying  hours,  availability  of  cargo,  the  location  of  the  cargo,  and  the  availability 
of  MOG  in  the  airlift  network,  among  other  thiugs.  In  addition,  a  planned  airlift  mission  includes 
all  the  bases  that  will  be  visited.  Conversely,  the  Markovian  transition  probability  matrices  used 
by  the  conceptual  analytical  model  are  constructed  on  the  assumption  that  the  probability  that  a 
customer  (aircraft)  moves  from  one  station  to  another  doesn’t  depend  on  its  prior  location.  However, 
given  that  the  TPFDD  requirement  remains  constant  for  all  replications,  a  constant  proportion  of 
the  cargo  at  each  starting  location  must  be  airlifted  to  each  of  the  destination  locations  it  is  not 
unreasonable  to  expect  some  amount  of  regularity  for  the  routing  of  airlift  missions  from  replication 
to  another.  Hence  these  realized  routing  proportions  could  logically  serve  as  approximations  of  the 
airlift,  planning  process.  Hence,  we  consider  the  conceptual  analytical  model  as  face  valid  and 
proceed  to  the  next  aspect  of  conceptual  analytical  model  validation:  input  and  output  matching. 

We  will  first  attempt  to  create  output  performance  measures  from  the  analytical  model  for 
estimating  EGA  and  ACR.  Note  that  both  of  these  performance  measures  are  functions  of  the 
number  of  aircraft  assigned,  the  cargo  delivered  every  day,  and  the  TPFFD  requirements.  For  the 
analytical  model,  we  observe  that  the  average  amount  of  cargo  delivered  per  day  by  each  type  of 
aircraft  is  a  function  of  the  average  throughput  of  aircraft  leaving  the  CONUS  air  bases  and  the 
average  aircraft  cargo  load. 

We  define  the  following  terms  in  order  to  compute  the  average  cargo  delivered  per  day.  Let 
B  =  {1,2,3}  be  the  set  of  CONUS  home  stations  such  that  JFK  lAP  is  base  1,  Charleston  AFB 
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is  base  2,  and  McGuire  AFB  is  base  3.  Also,  let  Y  =  {C,  M}  be  the  set  of  aircraft  types  in  the 
scenario  so  that  C  represents  GRAF  aircraft  and  M  is  military  aircraft.  Then  N'^  =  ^m) 

represents  the  number  of  GRAF  and  military  aircraft  in  the  scenario  respectively  where 

—  A^c-141  +  Ac-s  +  Ac-17  (7.32) 

is  the  sum  of  the  three  types  of  military  aircraft  used  in  the  corresponding  AFM  scenario.  The 
average  throughput  per  hour  for  base  i  €  B  as  calculated  by  the  analytic  model  is  given  by 
Ai  =  (Af ,  Xf^y  for  GRAF  and  military  aircraft  respectively.  Since  aircraft  leaving  their  home 
station  will  only  fly  to  another  home  station  if  they  are  going  to  pick-up  cargo,  we  consider  only 
those  aircraft  flying  to  bases  other  than  the  other  home  station  as  carrying  cargo.  Thus,  to  calculate 
cargo  throughput,  let  pfj  represent  the  probability  that  a  GRAF  aircraft  leaving  base  ie  B  will 
fly  to  base  j  €  B,  j  i  and  pA,  represents  the  same  probability  for  military  aircraft.  Thus  the 
probability  that  an  aircraft  of  type  y  gY  leaves  base  i  to  deliver  cargo,  P^{i),  is  given  by 

Pl(i)  =  1  -  y^Y,i€B  (7.33) 

jeB 

j:^i 

Then  the  average  throughput  per  hour  for  aircraft  of  type  y  E  Y  from  station  i  €  B  delivering 
cargo,  A^,  is  found  by 

Ar  =  An^’^(i)}.  yeY,i€B  (7.34) 

If  we  let  Gcrafj  Gc-i4i,  Gc-s,  and  Gc-i?  be  the  average  cargo  load  in  tons  for  each  type  of 
aircraft  as  realized  by  AFM,  we  can  compute  the  average  tons  of  cargo  delivered  every  hour  by 
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aircraft  type  C  for  station  i  €  B,  hy 


7f  =  Af  (Gcraf)  1  i€  B  (7.35) 

and  for  aircraft  type  M  by  conditioning  on  the  each  type  of  military  aircraft  given  by 

=  Af  (ATj^)  ^  [Ac-ui  Nc-s  A/c_i4i]  [C?c-i4i  Gc-s  Gc-it]^  >  i€  B  (7.36) 

Thus,  the  average  tons  of  cargo  delivered  per  day  as  computed  by  the  anal3ftical  model,  F^,  is  given 
by 


=  (7-37) 

yeY  ieB 

Given  the  average  tons  of  cargo  dehvered  per  day,  we  now  define  the  formulas  for  computing 
the  analytical  performance  measures  ECA^  and  ACR^.  To  compute  these  performance  measures, 
we  must  determine  the  amoimt  of  cargo  that  is  delivered  each  day  of  a  scenario.  In  AFM,  this  is 
a  straightforward  process.  We  need  only  record  the  values  realized  during  each  realization  of  the 
simulation  model.  For  the  conceptual  analytical  model,  it  is  not  a  straightforward  process.  First 
of  all,  the  procedure  outlined  above  provides  us  with  an  expected  value  for  the  amount  of  cargo 
dehvered  each  day,  F"^,  under  steady  state  assumptions  and  doesn’t  provide  us  with  the  probabihty 
distribution  of  that  value.  Further,  the  amount  of  cargo  “moved”  by  the  analytical  model  is  not 
constrained  by  a  TPFDD.  Therefore,  it  is  possible  to  compute  a  value  of  F"^  that  exceeds  the 
amount  of  cargo  available  for  movement  by  the  TPFDD  on  any  given  day.  Despite  these  inherent 
problems,  we  propose  a  method  for  estimating  ECA^  and  ACR^  using  the  analytical  model. 

We  make  two  comments  about  our  derivation  of  ECA^  and  ACRA  before  we  begin.  The 
first  comment  is  that  these  formulas  were  developed  in  an  iterative  process  by  comparing  analytical 
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model  results  for  several  proposed  formulas  and  methods  to  the  results  of  AFM  at  three  test 
design  points.  In  order  to  save  space  and  time  we  only  present  our  final  results.  Our  second 
comment  is  that  there  are  several  possible  approaches  that  an  analyst  could  use  to  approximate 
these  performance  measures.  We  are  in  a  diflBlcult  situation  in  that  we  are  attempting  to  use  an 
analytical  model  that  computes  mean  performance  measures  based  on  assumptions  of  steady  state 
operation  to  approximate  the  behavior  of  a  terminating  simulation  model.  We  have  developed  an 
approach  that  we  find  provides  satisfactory  results  for  our  purposes. 

In  determining  ECA^  and  ACR^  we  adjust  the  TPFDD  requirements  in  order  to  mimic 
the  mean  performance  measures  made  available  by  the  analytical  model.  We  first  assume  that 
the  amount  of  cargo  made  available  for  airlift  on  the  day,  V^{k)  is  divided  equally  across  the 
16  days  defined  in  the  TPFDD  and  the  required  delivery  amoimts,  c^{k)  are  also  divided  equally 
between  the  5^^  and  21^*  day  of  the  plan,  with  the  following  exception.  It  became  apparent  that 
the  amount  of  cargo  delivered  in  the  first  two  days  during  an  AFM  realization  is  always  much  less 
than  that  observed  over  the  remainder  of  the  replication.  Based  on  this  observation,  we  use  the 
observed  values  of  the  AFM  replications  to  set  the  availability  of  cargo  for  the  first  two  days.  Thus, 
for  n  replications,  we  set  the  mean  cargo  available  in  the  analytical  model  for  days  1  and  2  by 


n 


v^il)  = 

3-1 

(7.38) 

V\2)  = 

(7.39) 

so  that  the  cargo  available  for  movement  in  the  analytical  model  is  given  by 


V^{k)  = 


Cje(21)-(y^(l)  +  r^(2)) 


14 


A:  =  3,4,...  ,16 


(7.40) 


In  a  similar  manner,  the  amount  of  cargo  that  is  required  for  delivery  each  day  for  the  analytical 
model,  c^(fc)  is  foimd  by 


\{k)  —  < 


0, 

^n(21) 

16  ’ 


A:  =  1,2, ...  ,5 
jb  =  6,7,...  ,21 


and  the  accumulated  cargo  delivery  requirements  are 


(7.41) 


C'n W  =  fe  =  1, 2, . . .  , 21  (7.42) 

i<k 

To  determine  the  amount  of  cargo  delivered  each  day  by  the  analytical  model  we  begin  by 
making  the  assumption  that  the  amount  of  cargo  delivered  on  day  A:  =  1, 2, . . .  ,  21  by  the  anal3d;ical 
model,  c^{k)  is  bounded  above  by  and  bounded  below  by  the  amount  of  cargo  available  for 
delivery.  In  AFM,  there  is  obviously  a  delay  between  the  time  that  cargo  is  loaded  onto  an  aircraft 
until  it  is  delivered.  The  result  of  this  delay  is  that  some  portion  of  the  cargo  delivered  on  any 
given  day  is  on-loaded  that  same  day  while  the  remainder  is  on-loaded  the  previous  day.  Based  on 
this  observation,  we  limit  the  cargo  delivered  on  the  same  day  it  is  on-loaded  to  r^/2.  Thus,  the 
cargo  delivered  on  the  first  day  of  the  scenario,  c^(l)  is  given  by 


c^(l)  =  mm{r^/2,  y^(l)} 


(7.43) 


We  track  any  cargo  from  the  first  day  that  is  not  delivered,  or  the  backlog  cargo,  6(1),  by 


6(i)  =  y^(i)-(4(i) 


(7.44) 


Then  for  the  rest  of  the  scenario,  we  determine  the  cargo  delivered  each  day  in  the  following  manner. 
First  we  find  the  amount  of  cargo  that  is  on-loaded  the  previous  day  for  delivery  on  the  current 
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day  represented  by  This  is  equal  to  the  backlog  cargo  from  the  previous  day,  up  to  the  mean 
cargo  throughput  Mathematically,  this  is  formulated  as 


Vj(fc)  =  min{6(fc-1),  r^},  fc  =  2,3,...  (7.45) 

where 

b{k)  =  b{k-l)-ip{k),  A:  =  2,3,...  (7.46) 

Next  we  determine  p{k),  the  remaining  capacity  to  on-load  and  deliver  cargo  on  the  same  day  given 
by 

p{k)  =  min  {r^  -  <fi{k),  r"^/2}  ,  A:  =  2, 3, . . .  (7.47) 

which  is  limited  by  the  amount  of  cargo  that  is  actually  made  available  for  on-loading  on  that  day 
so  that  the  actual  cargo  on-loaded  and  delivered  on  the  same  day  is  given  by 

6{k)  =  min  {p(A:),  V^{k)}  ,  A:  =  2, 3, . . .  (7.48) 

Thus  we  can  now  compute  the  cargo  delivered  on  day  k  by 

4  (k)  =  <fi{k)  +  Sik),  A:  =  2, 3, . . .  (7.49) 

As  before,  the  accmnulated  delivered  cargo  by  each  day  is  given  by 

fe  =  l,2,...  (7.50) 

i<k 
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This  process  continues  until  the  analytical  model  TPFDD  cargo  requirement  is  completely  delivered. 
The  day  that  the  TPFDD  is  closed  out,  T^,  is  given  by 

T^  =  k3{k  =  l,2,...\C^{k)  =  Cr{21)  and  -  1)  <  Cr(21)}  (7.51) 


The  amount  of  time  required,  in  hours,  to  deliver  all  cargo,  is  given  by 


r^  =  24 


J 


(7.52) 


Given  the  values  of  C^{k)  and  computed  in  the  manner  described  above,  we  estimate 
ECA^  and  ACR^  as  follows.  ECA^  is  found  by 


"pA 


ECA^  =  ~Y.{^Dii)-Cim 


(7.53) 


2=1 


To  compute  ACR^  we  first  find  the  ratio  between  the  amount  of  time  required  to  close  out  the 
TPFDD  to  the  amount  of  time  the  TPFDD  requires  all  cargo  to  be  delivered  by,  which  is 
defined  as 


'  D 


24-21 


(7.54) 


and  the  ratio  of  the  total  cargo  requirement  to  the  accumulated  cargo  delivered  by  the  last  required 
day  given  by 


■A_CRi21) 


C^{21) 


(7.55) 
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Then  the  estimated  ACR  is  given  by 


ACR^  =  I 


if  <  21 


A{D^){M^) 

A{D^){M^)  +  if  >  21 


(7.56) 


where  is  the  average  cargo  load  over  all  aircraft  types  foimd  by 


qA  _  1  j-jyA  JVc-i41  1^0-141]  [^CRAF  ^?C-141  Gc-5  Gc-l?]^  (7.57) 


Therefore,  based  on  these  calculations,  we  have  a  method  for  determining  the  same  output  perfor¬ 
mance  measures  using  the  analytical  model  that  are  estimated  by  AFM  in  the  RSM  study. 

We  now  consider  how  the  four  treatment  levels  are  adjusted  in  the  conceptual  ^alytical 
model.  The  four  treatment  levels  in  AFM  are  the  number  of  each  of  the  four  aircraft  types  in  the 
aircraft  fleet.  For  the  conceptual  analytical  model,  we  can  also  input  the  number  of  aircraft  in  the 
fleet.  However  we  have  only  two  inputs,  the  number  of  CRAF  aircraft,  iV^,  and  the  number  of 
military  aircraft,  which  is  the  total  of  all  C-141,  C-5,  and  C-17  aircraft  in  the  AFM  scenario, 
when  computing  the  closed  queueing  network  using  the  MVA  algorithm.  Therefore,  we  can  adjust 
the  treatment  levels  in  the  conceptual  anal3rtical  model  to  match  the  treatment  levels  in  AFM  as 
Ulustrated  in  Table  7.18.  The  obvious  difference,  and  problem,  is  that  it  appears  that  there  are 
only  8  distinct  design  points  out  of  the  16  different  design  points.  At  this  level  of  analysis,  the 
analytical  model  doesn’t  meet  the  requirement  of  unique  treatment  mappings. 

Other  inputs  to  the  conceptual  analytical  model,  including  the  formulas  for  computing  the 
output  performance  measures,  do  uniquely  determine  each  of  the  16  experimental  design  points. 
For  example,  the  realized  mean  service  times  and  routing  proportions  from  AFM  and  then  input  to 
the  conceptual  analytical  model  are  determined  by  the  aircraft  mix.  These  different  input  values 
will  contribute  to  differentiate  each  of  the  experimental  design  points  in  the  analytical  model.  More 
importantly,  the  mean  cargo  throughput  per  hour  computed  using  Equation  (7.36)  is  a  function  of 
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Table  7.18  Initial  analytical  model  RSM  uncoded  2^  factorial  design. 


Design 

Point 

Number  of  Aircraft 

CRAF 

Military 

1 

30 

190 

2 

30 

170 

3 

30 

170 

4 

30 

150 

5 

30 

170 

6 

30 

150 

7 

30 

150 

8 

30 

130 

9 

20 

190 

10 

20 

170 

11 

20 

170 

12 

20 

150 

13 

20 

170 

14 

20 

150 

15 

20 

150 

16 

20 

130 

the  different  average  cargo  load  for  each  type  of  aircraft  and  the  probability  that  each  particidar 
type  of  military  aircraft  delivers  the  cargo.  That  probability  is  a  function  of  the  actual  number  of 
each  type  of  military  aircraft  as  deffned  by  the  AFM  experimental  design  point.  Hence,  the  output 
of  the  conceptual  analytical  model  is  a  function  of  each  unique  AFM  experimental  design  point 
and  can  be  adjusted  in  the  same  manner  as  that  of  AFM. 

We  have  validated  the  conceptual  analytical  model  to  AFM  using  the  conceptual  analytical 
model  validation  process  outlined  in  Section  6.3.1  and  Figure  7.1.  We  next  address  surrogate 
search  operational  validity  by  performing  the  ACV  method  at  each  of  the  initial  experimental 
design  points.  We  again  decline  to  describe  the  computerized  model  verification  process  other  than 
to  say  we  did  verify  our  computerized  model.  In  this  case  the  process  was  fairly  simple  since  we  are 
using  a  more  simplified  version  of  the  same  MYA  algorithm  used  in  the  Psuedo-BRACE  application 
above. 


7. 3. 5. 2  Surrogate  Search  Operational  Validity,  We  determine  surrogate  search  op¬ 
erational  validity  by  using  the  two-step  process  described  in  Section  6.3.3  and  outlined  in  Figure 
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7.2.  The  first  step  is  to  replicate  the  AFM  simulation  model  at  each  of  the  experimental  design 
points  in  order  to  estimate  the  performance  measures  using  the  ACV  method.  The  results  of  the 
ACV  method  are  then  analyzed  to  assess  the  predictive  ability  of  the  analytical  model.  During  the 
second  step  of  the  process  we  estimate  response  surfaces  for  both  models  in  order  to  compare  the 
two  models  across  the  entire  experimental  design  space.  If  both  of  these  steps  return  satisfactory 
results  the  analytical  model  meets  operational  validity  and  can  then  be  used  to  perform  a  surrogate 
search. 

We  generate  10  replications  of  AFM  at  each  of  the  16  experimental  design  points  in  order 
to  initiate  the  RSM  study  and  perform  ACV  design  point  validation.  We  use  the  ACV  method 
to  estimate  the  average  early  accumulated  cargo  per  day  per  aircraft,  ECA,  and  the  aircraft-cargo 
ratio,  ACR.  We  then  analyze  the  results  for  each  performance  measure  in  turn  in  accordance  with 
Figure  7.2  to  assess  ACV  design  point  validity  using  the  criteria: 

1.  ACV  linear  regression  model  is  appropriate 

(a)  “Significant”  variance  reduction 

(b)  Linear  scatter  plot 

2.  ^  «  1.0 

S.Y^^Z 

The  variance  reduction  achieved,  0,  AFM  estimated  ECA,  ECA  ,  the  mean  of  the  ACV, 
ECA^^  the  relative  difference  between  ECA^  and  ECA^  ^  and  the  observed  variance  of  ECA  at 
each  of  the  16  initial  design  points  are  listed  in  Table  7.19.  Since  variance  reduction  is  not  achieved 
at  all  design  points,  the  estimated  simulation  output  reported  is  not  the  ACV  controlled  response. 
At  first  glance,  it  doesn’t  appear  that  the  anal3dical  model  achieves  ACV  design  point  validation. 
At  only  9  out  of  the  16  design  points  does  the  ACV  actually  reduce  variance  and  at  only  one 
design  point  (12)  does  the  level  of  variance  reduction  approach  a  “significant”  level.  Further,  none 
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of  the  estimated  values  of  P  can  be  cotisidered  to  be  equal  to  approximately  one,  and  the  relative 
difference  between  ECA^  and  ECA^  is  fairly  large.  However,  further  analysis  using  some  of  the 
alternative  criteria  from  section  6.3.3. 1  reveals  a  closer  correspondence  between  the  two  models 
than  initially  thought. 


Table  7.19  ACV  results  for  EC  A  at  all  design  points. 


Early  Accumulated  Cargo  per  Day  per  Aircraft 

Design 

Point 

Variance 
Reduction  (%) 

ECA^ 

ECA^ 

Relative 
Difference  (%) 

Var  [eCA^) 

1 

Kii&il 

ihmibs 

2 

-12.31 

BlIiTgl 

19.72 

HKSISHi 

3 

-11.46 

■ilIEl 

27.19 

mESEEHH 

4 

00 

1 

30.03 

HHEESHi 

5 

-6.14 

19.82 

WKEMSKM 

6 

-12.49 

gmiiKi 

19.52 

WKSSSSII^M 

7 

1.54 

Blllll 

■SlilBl 

WKSSS3KM 

8 

-12.49 

ElMllltl 

mmm 

9 

9.84 

32.35 

26.05 

19.47 

10 

-8.83 

35.61 

28.42 

11 

-5.25 

35.75 

HESSIBH 

12 

28.95 

■tllrgl 

IHSMSI 

13 

■iUCT 

■tliirliii 

19.77 

0.0004 

14 

-12.34 

Tina 

0.0022 

15 

2.65 

■US 

39.98 

16 

-11.62 

44.71 

We  examine  two  aspects  of  the  results  in  Table  7.19  to  further  assess  analytical  model  vahdity 

_ g 

at  the  design  point  level.  First  of  all,  the  estimated  variance  of  EC  A  is  extremely  small  compared 

_ g 

to  the  observed  value  of  EGA  which  makes  the  task  of  variance  reduction  very  difficult.  Given  the 
relatively  small  amount  of  observed  variance  it  is  not  surprising  that  the  ACV  is  not  very  successful 
at  reducing  variance.  Therefore,  the  results  for  variance  reduction  achieved  and  for  the  value  of 
P  may  not  be  fair  indicators  of  surrogate  performance.  Secondly,  the  relative  difference  between 
the  means  of  the  two  models  is  approximately  identical  to  20%  at  every  design  point,  indicating 
that  although  the  model  outputs  are  not  approximately  the  same,  a  very  simple  adjustment  can 
be  made  to  ECA^  in  order  for  the  models  to  be  approximately  equal.  A  simple  calculation  yields 
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that  on  average  EGA  is  approximately  1.25  times  larger  than  EGA  at  each  of  the  design  points. 

_ y4 

Thus,  we  can  compute  an  “adjusted”  analytical  model  output,  EGAj^^j  by 


ECA%  =  l.2bECA^  «  ECA^  (7.58) 

Table  7.20  lists  the  observed  ratio  of  EGA  to  EGA  at  each  design  point,  the  mean  of  that 
ratio,  the  value  of  EGAj^^j^  and  the  relative  difference  between  EGA  and  EGAj^^j  at  each  design 
point.  Obviously,  the  observed  small  relative  difference  between  the  two  models  using  the  adjusted 
analytical  model  output  indicates  that  the  adjusted  analytical  model  is  an  excellent  predictor  of 
the  AFM  output  for  EGA  at  each  design  point. 


Table  7.20  Adjusted  ACV  results  for  EGA  at  all  design  points. 


Design 

Point 

^^9 

B 

Relative 
Difference  (%) 

1 

30.90 

25.01 

1.235 

31.27 

-1.20 

2 

33.87 

mkMm 

1.246 

■HBSmHiil 

3 

34.00 

0.02 

4 

37.49 

30.03 

1.248 

5 

34.02 

27.28 

1.247 

6 

1.243 

7 

MlWM 

1.257 

8 

42.08 

33.64 

1.251 

42.05 

0.05 

9 

32.35 

26.05 

1.242 

32.57 

-0.66 

10 

1.253 

11 

1.257 

12 

31.51 

1.259 

0.72 

13 

28.70 

1.246 

-0.29 

14 

1.253 

15 

1.262 

16 

44.71 

35.77 

1.250 

44.71 

0.01 

Mean  Ratio 

1.250 

_ Q  _ 

We  also  examine  the  two-dimensional  scatter  plots  between  EGA  and  EGA  provided  in 

Figure  7.10.  The  data  from  four  design  points  are  included  in  each  scatter  plot.  The  scatter  plots 

indicate  that  EGA  output  from  both  models  are  clustered  within  small  neighborhoods  of  their 

_ ^ 

sample  mean.  This  confirms  our  observation  that  there  is  little  A^iance  hi  EC  A  and  that  if  we 
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make  an  appropriate  adjustment,  the  analytical  model  provddes  an  excellent  prediction  of  EGA  at 
each  of  the  design  points.  Therefore,  based  on  the  alternative  criteria,  we  consider  the  first  step  in 
surrogate  search  operational  validity  to  be  complete  for  the  estimators  of  EGA. 


ACV 


ACV 


Figure  7.10  Two-way  scatter  plots  for  EGA. 


We  now  turn  our  attention  to  the  aircraft  cargo  ratio,  AGR,  results.  Table  7.21  contains 
the  observed  variance  reduction  achieved,  AFM  estimated  AGR,  AC R  ,  the  mean  of  the  AGV, 
ACR^^  the  relative  difference  between  ACR^  and  ACR^,  and  the  observed  variance  of  ACR  at 
each  of  the  16  initial  design  points.  Again,  we  don’t  meet  the  normal  criteria  for  AGV  design  point 
validation.  In  this  case,  variance  reduction  is  achieved  at  only  4  of  the  16  design  points  and  P  is 


approximately  equal  to  1  at  only  5  different  design  points.  However,  the  relative  difference  between 
'ACR^  and  AUR^  is  not  that  large  and  relatively  constant  at  an  average  of  3.5  percent. 


Table  7.21  ACV  results  for  ACR  at  all  design  points. 


Aircraft-Cargo  Ratio 

Design 

Point 

Variance 
Reduction  (%) 

ACR^ 

ACR^ 

Relative 
Difference  (%) 

1 

-11.47 

-1.23 

178.47 

■Iritlia 

0.118 

2 

-9.83 

■Em 

3.74 

0.042 

3 

26.96 

-8.59 

162.44 

3.57 

0.085 

4 

-11.96 

mss 

3.72 

5 

-11.53 

-1.23 

■raMiiliil 

3.41 

6 

-1.38 

-2.52 

■EIWiEM 

0.033 

7 

6.83 

-6.83 

146.08 

■CTigl 

3.42 

8 

-1.91 

130.34 

1^1 

3.63 

9 

29.92 

-6.52 

170.49 

3.60 

-12.49 

■EIQI 

3.42 

11 

-11.72 

3.33 

12 

-10.01 

138.40 

133.61 

3.47 

13 

-6.82 

2.90 

154.39 

3.59 

14 

-1.00 

3.78 

15 

10.48 

-9.60 

133.41 

3.36 

0.112 

16 

HIHIiSlHI 

-12.09 

121.68 

117.81 

3.18 

0.172 

As  with  EGA,  we  assess  the  ACV  design  point  validity  of  the  analytical  model  by  considering 

_ ^ 

the  alternative  criteria.  First  we  note  that  the  observed  variance  of  ACR  is  small  compared 
to  its  observed  value  making  variance  reduction  very  difficult  to  achieve.  So  we  again  disregard 
the  variance  reduction  results  and  focus  on  the  relative  difference  between  the  two  models.  As 
previously  noted,  the  relative  difference  between  the  observed  means  of  the  two  models  is  fairly 
constant  at  3.5  percent.  We  adjust  ACR^  as  before  by  first  finding  the  sample  mean  over  all  16 
design  points  of  the  ratio  of  ACR  to  ACR  which  is  approximately  equal  to  1.037.  Then  the 
adjusted  analytical  model  output  is  given  by 


ACR^^  =  1.037ACir^  «  ECAs 


(7.59) 
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We  present  the  results  of  the  above  calculations  in  Table  7.22  at  each  design  point  which  indicate 

_ g 

that  the  adjusted  analytical  model  is  an  excellent  predictor  for  ACR  at  each  design  point. 


Table  7.22  Adjusted  ACV  results  for  ACR  at  all  design  points. 


Design 

Point 

ACR^ 

ACR^ 

i— 

ACRi^ 

Relative 
Difference  (%) 

1 

1.037 

mmmm 

2 

■rritifiB 

1.039 

0.21 

3 

162.44 

1.037 

0.04 

4 

146.65 

■iraKai 

0.19 

5 

162.05 

156.52 

1.035 

6 

1.038 

■BfHbliM 

0.17 

7 

1.035 

8 

1.038 

WKtKESStKM 

iiEiim 

1.037 

0.07 

HOB! 

154.24 

148.96 

1.035 

■lifgkM 

11 

154.15 

149.01 

1.034 

■WBIbI 

12 

1.036 

HIH&SI 

13 

1.037 

■liffciiM 

HHKSBH 

14 

133.37 

1.039 

15 

138.05 

1.035 

16 

121.68 

■iWzf 

1.033 

-0.37 

Mean  Ratio 

1.037 

The  two-dimensional  scatter  plots  between  ACR  and  ACR  are  presented  in  Figure  7.11 
with  data  from  four  design  points  included  in  each  scatter  plot.  We  observe  that  the  output 
from  both  models  are  clustered  within  small  neighborhoods  of  their  respective  sample  means  as  a 
resulting  in  the  small  observed  variance  for  EC  A  .  Further,  the  plots  confirm  our  claim  that  by 
adjusting  the  analytical  model  output  by  Equation  (7.59),  the  analytical  model  meets  the  alternative 
criteria  for  surrogate  search  operational  validity  at  the  design  point  level  for  ACR. 

Given  that  the  analytical  model  has  passed  the  first  step  of  surrogate  search  operational 
validity  we  assess  the  second  step — ^response  surface  validity.  We  begin  by  estimating  the  response 
surface  generated  by  both  AFM  and  the  anal5d;ical  model.  The  estimated  coefiicients  for  the 
response  surfaces  for  both  EGA  and  ACR  are  listed  in  Table  7.23.  For  the  AFM  responses  we  also 
include  their  estimated  variance.  The  AFM  coefficients  are  estimated  using  all  160  uncontrolled 
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ACV  ACV 

Figure  7.11  Two-way  scatter  plots  for  ACR. 

responses  for  both  performance  measures.  The  analytical  coefficients  are  estimated  by  using  the 
least  squares  approximation  method  at  each  of  the  16  design  points.  At  each  design  point,  we 
calculate  the  analytical  responses  using  the  sample  means  of  the  appropriate  service  times  and 
routing  probabilities  generated  by  AFM.  Then  we  calculate  each  analytical  response  using  the 
adjusted  responses  as  outlined  in  Equations  (7.58)  and  (7.59). 

To  determine  response  surface  validity  we  compare  the  gradients  of  each  estimated  response 
surface  and  the  estimated  responses  at  the  center  of  the  experimental  design.  We  compare  the 
gradient  vectors,  and  by  determining  costp  where  ^0  is  the  angle  form^  between  the  two 
gradients.  If  cos  ^  «  1  then  the  two  gradients  point  in  similar  directions  indicating  that  both 
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Table  7.23  AFM  and  analytical  model  response  surface  parameter  estimates. 


AFM  Estimated  Parameters 

Response 

Sf 

H 

bl 

ECA^^ 

36.96 

-0.97 

-2.00 

-1.99 

-1.88 

0.00098 

ACR^ 

150.33 

4.07 

8.12 

8.10 

7.94 

0.0045 

Analytical  Model  Estimated  Parameters 

Response 

bt 

bt 

ECA^ 

36.95 

-0.91 

-1.97 

-1.87 

-1.87 

ACR^ 

150.37 

3.99 

8.08 

8.00 

8.02 

response  surfaces  are  oriented  approximately  the  same  in  the  design  space.  Then  if  the  response 
at  the  design  center  are  also  approximately  equal,  the  analytical  model  provides  good  predictions 
of  AFM  across  the  design  space.  The  observed  values  of  cos  V?,  the  estimated  responses  at  the 
design  center,  and  their  relative  difference  in  percentage  are  detailed  in  Table  7,24. 


Table  7.24  AFM  response  surface  results. 


Response 

Gradient  Angle 

Design  Center 

'll) 

COsV^ 

m 

Relative 
Difference  (%) 

\m^m 

1.57 

0.9996 

36.96 

0.020 

ACR 

0.60 

0.9999 

150.33 

150.37 

-0.029 

The  results  of  the  response  surface  comparisons  listed  in  Table  7.24  indicate  the  response 
surfaces  are  approximately  the  same.  The  gradient  an^es  for  both  responses  are  yield  cosines 
nearly  equal  to  one,  and  the  observed  design  centers  both  have  a  relative  difference  of  less  than 
tenth  of  a  percent.  Therefore  based  on  these  results  and  the  results  of  the  ACV  design  point 
comparisons  we  conclude  that  the  analytical  model  meets  the  surrogate  search  operational  vahdity 
criteria  and  can  then  be  used  to  conduct  a  surrogate  search  for  the  RSM  studies.  The  results  of 
the  surrogate  search  are  described  below. 


7.5.5.5  AFM  Surrogate  Search  Results.  We  begin  the  surrogate  search  procedure 
by  examining  the  gradient  vectors  estimated  during  the  response  surface  validation  process.  We 


first  find  the  unit  gradient  vectors,  and  v^cr’  estimated  by  the  AFM  responses  ECA^  and 

ACR^  respectively,  which  are 


-0.2795 

0.2804 

-0.5674 

0.5587 

'^ACR  — 

-0.5627 

j  '^ECA  — 

0.5575 

-0.5322 

0.5464 

We  observe  that  nfcA  ^  ~'^ACR  S^al  is  to  maximize  EC  A  and  to  minimize  ACR. 

Thus,  to  optimize  EC  A  we  must  take  steps  in  the  direction  of  and  to  optimize  ACR  we  must 

take  steps  in  the  direction  of  — Therefore  we  can  search  for  the  optimal  values  of  EC  A  and 
ACR  by  proceeding  in  a  single  direction  defined  by  either  or  —v^cr- 

To  determine  the  actual  surrogate  search  path  and  step  size  we  begin  with  two  observations. 
The  first  observation  is  that  we  can  only  adjust  the  inputs  to  the  analytical  model  at  discrete 
levels  that  correspond  to  the  number  of  aircraft  in  the  scenario.  The  second  observation  is  that 
the  gradient  terms  that  correspond  to  the  coded  military  aircraft  inputs,  02,03,  and  04,  are  all 
approximately  twice  the  size  of  the  term  that  corresponds  to  the  coded  CRAF  aircraft  input,  0i. 
Therefore  for  simplicity  we  define  the  surrogate  search  steepest  ascent  gradient,  g**,  by 


-1.0 

-2.0 

-2.0 

-2.0 


(7.61) 


Now  we  determine  the  smallest  step  size  possible  that  will  result  in  input  factors  that  correspond 
to  integer  values  for  the  uncoded  aircraft  inputs.  Since  the  gradient  terms  for  the  military  aircraft 
are  twice  the  size  of  the  gradient  term  for  the  CRAF  aircraft  we  want  to  find  the  step  size  that  wiU 
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reduce  the  number  of  CRAF  aircraft  by  one  for  each  step.  The  appropriate  coded  step  size,  A*®,  is 
found  by  setting  0i  =  1  in  equation  7.28,  where  0i  is  the  number  of  CRAF  aircraft,  and  solving 
for  Xi  =  Thus  the  surrogate  search  step  size  is  given  by 

^  =  1  =»  A*®  =  0.2  (7.62) 


Based  on  this  coded  step  size  and  surrogate  search  steepest  ascent  gradient,  the  coded  and  uncoded 
surrogate  search  points  are  listed  in  Table  7.25.  For  those  coded  treatment  levels  that  translate  to 
negative  aircraft  levels,  the  aircraft  level  is  set  to  zero. 


Table  7.25  Surrogate  search  steps. 


Design 

Point 

lireatment  ] 
Coded  1 

Levels 

Uncoded 

01 

02 

03 

04 

Xi 

^2 

^3 

X4 

1 

-0.2 

-0.4 

-0.4 

-0.4 

24 

46 

56 

46 

2 

-0.4 

-0.8 

-0.8 

-0.8 

23 

42 

52 

42 

3 

-0.6 

-1.2 

-1.2 

-1.2 

22 

38 

48 

38 

4 

-0.8 

-1.6 

-1.6 

-1.6 

21 

34 

44 

34 

5 

-1.0 

-2.0 

-2.0 

-2.0 

20 

30 

40 

30 

6 

-1.2 

-2.4 

-2.4 

-2.4 

19 

26 

36 

26 

7 

-1.4 

-2.8 

-2.8 

-2.8 

18 

22 

32 

22 

8 

-1.6 

-3.2 

-3.2 

-3.2 

17 

18 

28 

18 

9 

-1.8 

-3.6 

-3.6 

-3.6 

16 

14 

24 

14 

10 

-2.0 

-4.0 

-4.0 

-4.0 

15 

10 

20 

10 

11 

-2.2 

-4.4 

-4.4 

-4.4 

14 

6 

16 

6 

12 

-2.4 

-4.8 

-4.8 

-4.8 

13 

2 

12 

2 

13 

-2.6 

-5.2 

-5.2 

-5.2 

12 

0 

8 

0 

14 

-2.8 

-5.6 

-5.6 

-5.6 

11 

0 

4 

0 

15 

-3.0 

-6.0 

-6.0 

-6.0 

10 

0 

0 

0 

16 

-3.2 

-6.4 

-6.4 

-6.4 

9 

0 

0 

0 

17 

-3.4 

-6.8 

-6.8 

-6.8 

8 

0 

0 

0 

18 

-3.6 

-7.2 

-7.2 

-7.2 

7 

0 

0 

0 

19 

-3.8 

-7.6 

-7.6 

-7.6 

6 

0 

0 

0 

20 

-4.0 

-8.0 

-8.0 

-8.0 

5 

0 

0 

0 

We  are  now  faced  with  the  problem  of  determining  the  inputs  to  the  analytical  model  at 
each  of  the  steps  along  the  path  of  steepest  ascent  (descent)  listed  in  Table  7.25.  The  method  for 
translating  the  aircraft  fleet  to  the  appropriate  settings  for  the  analytical  model  as  described  above 
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in  Section  7.3.51  still  holds  for  the  surrogate  search.  The  problem  is  determining  the  appropriate 
settings  f3r  the  aircraft  routing  probabilities  and  the  service  (waiting)  times  at  each  of  the  air  bases. 
For  the  purposes  of  generating  an  ACV  the  observed  sample  means  from  the  AFM  replications  for 
each  of  the  described  values  are  used  as  inputs  to  the  analytical  model.  When  performing  a 
surrogate  search,  these  AFM  outputs  are  obviously  not  available.  To  set  these  values,  we  have 
essentially  two  choices.  The  first  choice  is  to  use  statistical  methods  to  predict  the  input  values 
based  on  the  observed  values  generated  during  the  initial  stages  of  the  RSM  study.  The  second 
choice  is  base  the  settings  on  an  imderstanding  of  the  processes  that  give  rise  to  the  value.  We  use 
a  combination  of  both  methods  as  described  below. 

The  analytical  model  inputs  are  defined  in  the  following  manner.  Let  the  probability  transition 
matrices  of  aircraft  routings  be  represented  by 


TT: 


TT- 


r(2,i) 


y(i,2) 

0 


TT; 


(1,13) 
^y(2,i3) 


A  A 

’''y(i3,i)  ^y(i3,2) 


,  Y&{C,  M} 


(7.63) 


where  TTy^^  is  the  steady  state  probability  that  an  aircraft  of  tjrpe  Y  will  fly  from  base  i  to  base 
3  with  Y  e  {C,  M}  representing  CRAF  and  military  aircraft  respectively.  Since  aircraft  don’t 
take-off  and  land  at  the  same  airfleld,  ~  ®  ~  2)  •  •  •  j  13.  Note  that  these  13  x  13 

matrices  are  transformed  into  matrices  of  size  up  to  169  to  represent  the  total  state  space  of  the 
model,  however,  each  Ily  completely  determines  the  size  and  values  of  each  of  the  larger  matrices. 
Therefore,  we  consider  only  11^  and  II)^  for  this  discussion.  We  let  the  mean  time  that  an  aircraft 
of  type  Y  spends  at  airbase  i,  i  =  1,2, . . .  ,  13  by  As  before,  the  number  of  each  type  of 

aircraft  are  symbolized  by  Nq  for  CRAF  aircraft  and  =  Nc-m  +  Nc—s  +  ^c-n  for  military 
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aircraft  with  Nc~iai:  5?  ^c—i7  ^he  nuiiiber  of  C-141,  C-5,  and  C-17  aircraft  respectively 

at  the  particular  design  point. 

In  a  similar  manner,  we  define  the  corresponding  terms  for  AFM  generated  values.  To  define 
the  observed  routing  probabilities  for  AFM  let  be  the  number  of  aircraft  of  type  Y  that 

fiy  from  airbase  i  to  airbase  j  {i,j  =  1, 2, . . .  ,  13)  during  AFM  repUcation  k  =  1, 2, . . .  Again, 
=  0  for  z  =  1, 2, . . .  ,  13.  The  total  number  of  departures  observed  at  base  i  for  aircraft  type 
Y  during  replication  k  is  then  given  by 

13 

^  i  =  lj2,...,13  &  =  1,2,  ...,n  (7»64) 

j=i 

so  that  the  routing  proportions  for  replication  k  are  given  by 


=  i,j  =  l,2,...,13  k  =  l,2,...,n  Ye{C,  M}  (7.65) 


with  the  estimated  routing  probabilities  computed  as 


^  *. J  =  1, 2, .  - .  ,  13 


k=l 


(7.66) 


The  two  routing  probability  matrices  are  then  given  by 


^Yil,2) 


JjrS 

^(1,13) 


TT- 


Y(2,l) 


—S  —S 

’'’y(i3,i)  ^y(i3,2) 


tt: 


y(2,i3) 


0 


,  Ye{C,  M} 


(7.67) 


To  estimate  the  mean  time  aircraft  spend  at  each  base  during  each  visit,  we  let  WY(i,i)ik) 
be  the  amoimt  of  time  the  departing  aircraft  of  type  Y  spends  at  base  i  during  replication  k. 
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Thus,  the  mean  time  an  aircraft  of  type  Y  spends  at  base  i  during  a  single  visit  for  rephcation  k  is 
computed  as 

-  (i)  (^) 

Wf (i)(fc)  =  £  WY(i,i){k),  i  =  l,2,...,13  k  =  l,2,..-,nYe{C,  M}  (7.68) 

so  that  we  can  estimate  the  mean  time  aircraft  of  each  type  spend  at  each  base  by 

Wy(,)  1,2,...  ,13  Ye{C,  M}  (7.69) 

k=X 

For  the  replications  generated  during  the  initial  first  order  design  of  experiment,  we  observe 
that  each  of  the  values  of  Ily  and  Wy  =  (W^y(i)j  W^y(2)>  •  •  •  5^y(i3)y  esch  of  the  16  design 
points  possess  little  variance,  except  for  some  of  the  waiting  times.  The  air  bases  that  exhibit  a 
large  variance  in  waiting  times  are  the  three  “home  stations”  of  Charleston  AFB,  McGuire  AFB, 
and  JFK  lAP.  Therefore,  we  will  use  the  “overall”  sample  means  of  ILy  and  Wy  (except  for  the 
three  home  stations)  as  inputs  to  the  analytical  model  for  the  purposes  of  performing  a  surrogate 
search. 

The  overall  sample  means  that  are  used  as  inputs  to  the  analytical  model  for  the  purposes 
of  performing  the  current  surrogate  search  are  defined  in  the  following  manner.  Let  Ily  (d)  be  the 
estimated  routing  probability  matrices  for  aircraft  type  Y  at  design  point  d  =  1, 2, . . .  ,16,  where 


j — 1,2,... , 


16 

,16 


d=l,2,...  ,16  Y€{C,  M} 


(7.70) 


Then  the  overall  sample  mean  for  the  routing  probability  matrices  are  given  by 


Y  e  {C,  M} 


(7.71) 
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Similarly,  let  Wy  (d)  be  the  vector  of  estimated  mean  waiting  times  at  air  bases  4, 5, . . .  ,13  (where 
Charleston  AFB  is  base  1,  McGuire  AFB  is  base  2,  and  JFK  lAP  is  base  3)  for  aircraft  of  type  Y 
at  initial  design  point  d.  The  overall  sample  mean  for  the  waiting  time  vector  is  then 

I  d=l  d=l  d=l  ) 

In  order  to  determine  the  waiting  time  inputs  for  the  home  stations  we  recall  that  aircraft 
wait  at  their  home  station  for  two  reasons.  First,  they  must  wait  in  a  FIFO  queue  for  an  available 
cargo  load.  Then  if  they  are  assigned  a  load,  they  might  be  grounded  a  designated  amount  of 
time  so  that  the  target  utilization  rate  for  their  type  of  aircraft  is  maintained  by  the  differential  or 
capping  procedures  within  AFM  [11].  We  can  compute  the  grounding  time  assigned  each  t3rpe  of 
aircraft  using  the  differential  use  rate  control  formula  used  by  the  differential  procedure  in  AFM  in 
order  to  estimate  the  mean  groimding  times.  However,  if  aircraft  must  wait  an  additional  amount 
of  time  for  a  mission,  this  formula  will  not  provide  an  accurate  estimate.  Analysis  of  the  initial 
design  region  indicates  that  more  aircraft  than  are  required  are  included  in  the  aircraft  fleet  and  as 
such,  the  waiting  times  at  each  of  the  home  stations  exceed  the  grounding  times  computed  using 
the  differential  use  rate  control  formula.  Therefore,  we  use  a  twofold  strategy  for  estimating  the 
waiting  times  at  these  bases.  First  we  use  linear  regression  at  the  16  design  points  with  the  aircraft 
fleet  as  the  independent  variables  and  each  air  base  waiting  time  as  the  dependent  variables.  We 
then  apply  the  estimated  response  functions  for  each  base  using  the  aircraft  fleet  at  the  appropriate 
surrogate  search  design  point.  We  also  compute  grounding  times  based  on  the  differential  procedure 
formula  and  the  aircraft  fleet  input  in  order  to  obtain  another  estimate  of  waiting  times  at  each 
base.  If  the  waiting  time  computed  using  the  linear  regression  approach  is  greater  (indicating 
aircraft  must  wait  for  a  mission)  we  use  that  input,  otherwise  we  use  the  waiting  time  computed 
using  the  differential  use  rate  formula. 
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The  home  station  waiting  time,  for  aircraft  of  type  Y  at  air  bases  i  =  1,2,3 

estimated  using  the  differential  use  rate  formula  is  computed  in  the  following  maimer.  Define  an 
aircraft  cycle  as  all  the  ground  and  flying  activity  that  occurs  to  an  aircraft  once  it  is  assigned  a 
cargo  load  imtil  it  returns  to  its  home  station  and  is  ready  to  accept  a  new  mission.  Then  let  gy 
be  the  expected  sum  of  all  ground  times  (servicing,  cargo  up-load  tod  off-load,  taxiing,  etc.  )  in 
hours  experienced  by  an  aircraft  of  t3rpe  y  G  {CRAF,  C— 141,  C-~5,C““17}  during  a  typical  cycle. 
Further  let  fy  be  the  expected  sum  all  flying  time,  in  horns,  accumulated  during  a  typical  cycle 
experienced  by  aircraft  of  type  y  £  {CRAF,  C  —  141,  C  —  5,  (7  —  17}.  Then  the  expected  utilization 
rate  (without  the  use  of  the  differential  procedure)  is  given  by 

Uy  =  y  €  {CRAF,  C  -  141,  C-5,C-17}  (7.73) 

fy+9v 

The  expected  additional  ground  time,  Sy,  or  slack  time,  added  to  the  end  of  each  cycle  for  aircraft 
of  type  y  as  calculated  by  the  differential  use  rate  formula  [11]  is  given  by 

=  (^y-'^y)  (fy  +9vf  ^  ^  {CRAF,  C  -  141,  (7  -  5,  C7  -  17}  (7.74) 

(24/3,) 

where  is  the  input  capability  rate  for  each  type  of  aircraft  listed  in  Table  7.16.  Using  observed 
values  of  Wy  and  lly  and  the  standard  servicing  times  listed  in  Table  7.16,  the  estimated  slack 
times  for  each  type  of  aircraft  are  hsted  in  Table  7.26. 

Table  7.26  Expected  slack  times  usiug  differential  use  rate  formula. 


Aircraft 

Inputs  (hrs) 

Slack  (hrs) 

m 

T 

s 

CRAF 

29.12 

10.75 

12.00 

12.56 

C-141 

33.40 

10.13 

12.20 

14.69 

C-5 

33.19 

15.75 

10.70 

16.77 

C-17 

31.80 

10.13 

15.30 

6.69 
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The  next  step  in  computing  the  expected  waiting  time  at  each  home  station,  (i)  (^)  is  to 
condition  on  the  probability  an  aircraft  experiences  its  slack  time  at  base  i  and  the  probability 
it  on-loads  cargo  at  the  base  L  We  begin  by  first  finding  the  expected  waiting  time  for  military 
aircraft  at  bases  1  and  2  (Charleston  AFB  and  McGuire  AFB).  Let  be  the  number  of  military 
aircraft  of  type  z  G  Mil  =  {C  -  141,  C  -  5,C  -  17}  assigned  base  *  =  1, 2  as  their  home  station. 
We  compute  p*(i),  the  probability  that  a  military  aircraft  of  type  z  has  home  station  i  by 


•  1  o 

~  ^  *  —  1) 2 
Z-iz  G  Mtl 


(7.75) 


where  Ng  is  the  number  of  military  aircraft  of  type  z  in  the  aircraft  fleet.  Next  we  compute  the 
probability  that  a  military  aircraft  at  station  i  is  a  home  station  aircraft  represented  by  Pi  by 


Pi  = 


p-Nz  +7rM(i,j)PjN„ 


i  —  1,2  j  —  1,2  i^j 


(7.76) 


where  Pi  =  [pc-ui(»),Pc-5(i),P£7-i7(i)]'  andN*  =  [Nc-ui,Nc-5,Nc-i7]' ■  Then  by  conditioning 
on  the  probability  that  an  aircraft  at  station  i  is  at  its  home  station,  the  expected  waiting  time 
based  on  the  dififerential  rate  use  formula  at  military  home  station  i  is  given  by 


W^M(i)(«)  =  Pi  {Pi  [S*  +  (1  -  tM(i,i))L.] }  +  (1  -  Pi)  {^MU,i)P'M  i  =  l,2  j  =  l,2  ij^j 

(7.77) 

where  =  [5c-~i4i,  Sc -5,8017]'  is  the  vector  of  computed  slack  times  shown  in  Table  7.26  and 
=  [Lc-141,  Lc-5,  Lc-ij]'  is  the  vector  of  standard  cargo  on-load  times  from  Table  7.16.  In  a 
similar  manner,  we  compute  the  expected  waiting  time  for  CRAF  aircraft  by 

^C(3)(^)  =  S  +  (1  -  fc(3,l)  '^C(3,2))^C  (7.78) 
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Given  the  methods  described  above  for  determining  the  appropriate  inputs  to  the  analytical 
model,  a  surrogate  search  is  performed  as  defined  in  Table  7.25.  The  results  for  both  EGA  and 
ACR  of  the  surrogate  search  are  provided  in  Table  7.27  using  and  ACR^.  For  brevity 

only  the  results  from  the  first  10  surrogate  search  steps  are  reported.  The  siurogate  search  results 
also  are  presented  graphically  in  Figures  7.12  and  7.13.  The  siurogate  search  produces  local  optima 
for  both  performance  measures  as  indicated  in  the  tables  and  graphs. 
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Surrogate  Search  Step 

Figure  7.13  Initial  surrogate  search  for  ACR. 

The  next  step  in  the  surrogate  search  procedure  is  to  validate  the  surrogate  search  results 
using  AFM  replications  at  what  will  presumably  be  design  points  in  the  next  series  of  experiments — 
if  the  surrogate  search  results  are  valid.  Due  to  the  number  of  factors  (four)  and  the  limited  nature 
of  the  surrogate  search  results  from  a  single  gradient  direction,  it  is  not  clear  if  the  observed 
local  optima  (if  they  are  predicted  correctly)  are  anywhere  near  the  global  optima  that  we  are 
searching  for.  Furthermore,  little  information  is  available  for  determining  an  appropriate  range 
for  each  of  the  treatment  levels.  Of  course  this  is  the  same  situation  we  would  be  in  if  we  didn’t 
use  the  surrogate  search  method  and  had  used  AFM  exclusively  to  explore  the  defined  gradient 
direction.  The  difference  is  that  with  the  surrogate  search  methodology,  we  can  rapidly  perform 
additional  surrogate  searches  to  gain  additional  information  about  the  predicted  behavior  of  AFM 
in  the  investigated  region.  Therefore  we  perform  additional  surrogate  searches  along  13  different 
gradients.  These  additional  gradients  point  in  the  same  general  direction  of  the  original  gradient — 
reducing  each  aircraft  level — ^however  the  proportion  of  aircraft  reduction  differs  for  each  graxlient. 
The  coded  gradients  for  the  additional  surrogate  searches  are  provided  in  Table  7.28, 
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Table  7.28  Additional  surrogate  search  gradients. 


Gradieni 

Coded  Treatment 

:  01 

02 

03 

04 

1 

-1.5 

-1.0 

-1.0 

-1.0 

2 

-1.0 

-1.0 

-1.0 

-1.0 

3 

-1.0 

-1.5 

-1.5 

-1.5 

4 

-1.0 

-2.0 

-2.0 

-2.0 

5 

-1.0 

-2.0 

-1.0 

-1.0 

6 

-1.0 

-1.0 

-2.0 

-1.0 

7 

-1.0 

-1.0 

-1.0 

-2.0 

8 

-2.0 

-2.0 

-1.0 

-1.0 

9 

-2.0 

-1.0 

-2.0 

-1.0 

10 

-2.0 

-1.0 

-1.0 

-2.0 

11 

-1.0 

-2.0 

-2.0 

-1.0 

12 

-1.0 

-2.0 

-1.0 

-2.0 

13 

-1.0 

-1.0 

-2.0 

-2.0 

We  perform  the  additional  surrogate  searches  and  provide  the  results  in  tables  7.29  and 
7.30.  We  list  the  10  largest  values  observed  for  ECA^^  and  10  smallest  values  observed  for 
over  the  14  surrogate  search  gradients,  with  the  aircraft  levels  that  generated  tbam  The 
sample  means  for  the  aircraft  inputs  and  the  performance  measures  are  also  included  in  the  tables. 
The  additional  gradients  have  identified  additional  aircraft  levels  that  produce  better  performance 
measure  responses  than  the  initial  surrogate  search.  Even  more  importantly,  we  have  a  better  idea  of 
the  predicted  range  of  each  aircraft  level  that  produce  desirable  performance  measure  responses — as 
long  as  the  surrogate  search  results  are  valid. 

Given  the  results  of  the  surrogate  searches  we  attempt  to  validate  the  results  using  AFM 
replications.  Rather  than  randomly  choosing  some  of  the  observed  surrogate  search  results  to 
validate,  we  first  design  a  new  set  of  experiments  to  estimate  the  next  response  surface  suggested 
by  the  surrogate  search  results.  We  then  validate  the  surrogate  search  by  replicating  the  AFM  model 
at  several  of  the  proposed  experimental  design  points.  If  the  results  are  validated,  we  complete  the 
proposed  design  using  the  results  of  the  validation  steps. 

We  begin  to  design  the  new  set  of  experiments  by  first  recognizing  the  need  for  a  design  that 
can  estimate  a  second  order  response  surface  and  by  setting  the  high,  center,  and  low  levels  of 
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Table  7.29  Largest  ECA^^j  observations  over  all  surrogate  searches  (with  sample  means). 


Aircraft  Levels 

ECAi,, 

CRAF 

0141 

05 

017 

19 

26 

48 

38 

53.92 

18 

22 

46 

— 

15 

30 

50 

mm 

17 

18 

WMm 

34 

20 

30 

■iM 

30 

13 

26 

48 

52.09 

30 

50 

mm 

52.08 

30 

1M 

40 

51.62 

21 

34 

wm 

42 

21 

52 

34 

20 

45 

35 

21 

34 

44 

34 

18 

36 

46 

36 

38 

mm 

38 

50.88 

40 

30 

50.83 

Mean  Values 

19 

31 

47 

37 

52.00 

each  of  the  aircraft  treatments.  We  also  recognize  that  the  aircraft  level  ranges  that  are  suggested 
by  the  surrogate  search  results  for  the  two  performance  measures  are  slightly  different.  However, 
we  dioose  to  design  a  single  set  of  experiments  to  estimate  response  smfaces  for  both  ECA^  and 
ACR^ .  Therefore  based  on  the  surrogate  search  results  we  decide  to  set  the  design  center  at 
x\  =  15,  OTj  =  25,  x%  =  40,  and  X4  =  30.  The  low  level  for  each  treatment  is  ^ven  by  xf"  =  5, 
X2  =  5,  ar3  =  25  and  =  15  and  the  high  levels  as  x^  =  25,  x^  =  45,  x^  =  55  and  =  45. 
Thus,  the  coded  treatment  levels  for  the  proposed  design  of  experiment  are  ^ven  by 


©i 


©^2 

©^3 

©i 


ari  -15 

10  ’ 

3  = 

a4  -  25 

20  ’ 

3  = 

1 

0 

15  ’ 

3  ~ 

a4-30 

15  ’ 

3  = 

c,H,L 

c,H,L 

c,H,L 

c,H,L 


(7.79) 

(7.80) 

(7.81) 

(7.82) 
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Table  7.30  Smallest  observations  over  all  surrogate  searches  (with  sample  means). 


Aircraft  Levels 

ACRi,, 

CRAF 

C-141 

C-5 

C-17 

15 

40 

30 

92.28 

7 

14 

42 

Kfl 

14 

6 

38 

16 

14 

32 

95.60 

9 

18 

mm 

34 

96.90 

17 

18 

28 

mm 

17 

18 

44 

mm 

17 

18 

44 

KB 

18 

32 

■kM 

11 

36 

TiTIHiM 

18 

22 

mMM 

26 

36 

■■1 

102.05 

9 

28 

38 

102.70 

5 

mm 

30 

14 

^^1 

28 

Mean  Values 

14  20 

40 

30 

99.01 

In  most  cases,  Box  and  Draper  recommend  a  central  composite  design  (CCD)  for  esti¬ 
mating  a  second  order  response  surface  [12].  A  CCD  is  a  full  factorial  design  augmented  with 
center  point  experiments  and  a  number  of  “axial”  or  “star”  coded  design  points  of  the  form 
(dzct,  0, . . .  ,0),  (0,  iba, ...  ,  0), . . .  ,  (0, 0, . . .  ,  ±Qj)  where  a  is  usually  chosen  equal  to  (n/)^/^  with 
rif  the  number  of  factorial  points  in  the  design.  In  our  case  then,  rif  =  16  so  that  a  =  2.  This 
presents  a  diflBlculty  though  because  Of  =  —2  and  Of  =  —2  translate  to  CRAF  and  C-141  aircraft 
levels  of  —5  and  *“15  which  are  of  course  physically  impossible.  Therefore  we  decide  to  construct 
a  Box-Behnken  design  that  requires  only  the  three  levels  already  defined  for  each  treatment  in 
order  to  estimate  a  second  order  response  surface  [12].  Although  not  all  Box-Behnken  designs  are 
rotatable,  the  Box-Behnken  design  for  4  variables  is  a  rotatable  second  order  design  that  requires 
only  27  trials  [12].  The  proposed  experimental  design  appears  in  Table  7,31. 

We  now  attempt  to  validate  the  surrogate  search  results  by  performing  AFM  replications. 
Examination  of  the  proposed  Box-Behnken  design  of  experiment  and  the  surrogate  searches  reveals 


7-69 


Table  7,31  Proposed  second  order  Box-Behnken  design  of  experiment. 


Design 

Point 

Coded 

Treatment  Levels 

Uncoded 

Treatment  Levels 

01 

02 

03 

04 

CRAF 

C-141 

C-5 

C-17 

1 

-1 

-1 

0 

0 

5 

5 

40 

30 

2 

1 

-1 

0 

0 

25 

5 

40 

30 

3 

-1 

1 

0 

0 

5 

45 

40 

30 

4 

1 

1 

0 

0 

25 

45 

40 

30 

5 

0 

0 

-1 

-1 

15 

25 

25 

15 

6 

0 

0 

1 

-1 

15 

25 

55 

15 

7 

0 

0 

-1 

1 

15 

25 

25 

45 

8 

0 

0 

1 

1 

15 

25 

55 

45 

9 

0 

0 

0 

0 

15 

25 

40 

30 

10 

-1 

0 

0 

-1 

5 

25 

40 

15 

11 

1 

0 

0 

-1 

25 

25 

40 

15 

12 

-1 

0 

0 

1 

5 

25 

40 

45 

13 

1 

0 

0 

1 

25 

25 

40 

45 

14 

0 

-1 

-1 

0 

15 

5 

25 

30 

15 

0 

1 

-1 

0 

15 

45 

25 

30 

16 

0 

-1 

1 

0 

15 

5 

55 

30 

17 

0 

1 

1 

0 

15 

45 

55 

30 

18 

0 

0 

0 

0 

15 

25 

40 

30 

19 

0 

-1 

0 

-1 

15 

5 

40 

15 

20 

0 

1 

0 

-1 

15 

45 

40 

15 

21 

0 

-1 

0 

1 

15 

5 

40 

45 

22 

0 

1 

0 

1 

15 

45 

40 

45 

23 

-1 

0 

-1 

0 

5 

25 

25 

30 

24 

1 

0 

-1 

0 

25 

25 

25 

30 

25 

-1 

0 

1 

0 

5 

25 

55 

30 

26 

1 

0 

1 

0 

25 

25 

55 

30 

27 

0 

0 

0 

0 

15 

25 

40 

30 

that  none  of  the  proposed  design  points  are  evaluated  during  the  surrogate  search.  Since  the 
analytical  model  can  be  computed  very  rapidly,  we  simply  pick  3  of  the  27  possible  design  points 
and  evaluate  the  analytical  model  at  those  points — using  the  surrogate  search  inputs — ^as  well 
as  performing  10  independent  replications  of  AFM  at  the  same  design  points.  Design  points  5 
(0  =  [0, 0,-1, -I]'),  9  (0  =  [0,0, 0,0]'),  and  8  (0  =  [0,0, 1,1]')  are  chosen  for  validation.  The 
results,  listed  in  Table  7,32,  indicate  that  the  surrogate  analytical  model  is  an  adequate  predictor 
of  AFM  output  behavior  at  the  tested  design  points.  Based  on  these  results,  we  decide  to  rely  on 
the  surrogate  search  results  and  perform  the  proposed  Box-Behnken  design  of  experiment. 


7-70 


Table  7.32  Surrogate  search  validation  results. 


Design 

Point 

1  Aircraft  Levels 

ECA 

ACR 

C-5 

C-17 

Surrogate 

AFM  1 

Surrogate 

AFM 

5 

15 

25 

mm 

-29.43 

317.60 

300.62 

9 

15 

25 

mM 

MtMl 

45.29 

101.91 

94.88 

8 

15 

25 

55 

45 

52.34 

47.67 

113.96 

113.78 

To  estimate  second  order  response  surfaces  for  both  of  the  performance  measures,  we  generate 
10  replications  of  AFM  at  the  remaining  24  design  points  defined  in  Table  7.31.  Based  on  these 
replications  we  estimate  response  surfaces  of  the  form 

E\y\  =  So  +  Si  01  +  S202  +  S303  +  S404  + 

Sll©!  +  S2202  +  S3303  +  S4404  + 

Si20102  +  Si30i03  +Si40i04  -f-  (7.83) 

S^30203  +  S24©2©4  + 

S34©3©4 


We  estimate  the  response  surfaces  without  the  use  of  an  ACV  due  to  the  small  observed  variance  of 
each  performance  measure  response  in  the  simulation  results.  The  estimated  parameters  and  their 
associated  standard  error  are  listed  in  Table  7.33.  For  the  ECA^  response  surface,  MSE  =  1.78, 
_  0.99,  and  the  regression  F  statistic  is  equal  to  2964.6.  The  same  values  for  the  ACR^  response 
surface  axe  MSE  —  253.67,  =  0.89,  and  the  F  statistic  is  142.5.  Thus,  in  both  cases  we  accept 

the  fitted  response  surfaces  as  statistically  adequate. 

We  now  analyze  the  fitted  response  surfaces  to  determine  the  optima  or  identify  a  new  space 

_ g 

for  further  experimentation.  First  we  note  that  the  maximum  observed  ECA  =  58.58  at  design 
point  21  (CRAF  =  15,  C-141  =  5,  C-5  =  40,  and  C-17  =  45)  and  the  minimum  observed  ACR^  — 
91.44  also  at  design  point  21.  We  now  attempt  to  locate  the  stationary  points  of  each  smdace,  if 
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Table  7.33  Second  order  response  surfaces  parameter  estimates. 


ECA^ 

ACR^ 

Pargmeter 

Standard 

Parameter 

Standard 

Parameter 

Estimate 

Error 

Error 

bo 

0.24 

■ehh 

2.91 

bi 

2.86 

1.45 

62 

2.31 

0.12 

1.45 

bz 

13.22 

■Em 

BEEEiH 

64 

IHS9I 

■■■■ 

611 

■m 

0.94 

2.18 

^22 

-1.90 

0.18 

5.56 

msm 

&33 

-6.63 

0.18 

KflH 

644 

-8.39 

0.18 

612 

-2.11 

HE^H 

mm 

^13 

-4.57 

HE&H 

■IEEH 

mmm 

■ESH 

■mi 

^24 

-9.31 

0.21 

■IBSi 

i>34 

-15.13 

0.21 

56.13 

2.52 

they  exist.  Recall  that  the  stationary  point  of  a  second  order  function  with  4  variables  is  given  by 


e. 


(7.84) 


where  b  =  (fei,  621  ^>3) ^>4)'  and 


bn 

\bi2 

I&13 

I*h4 

1^12 

i>22 

1^3 

|*»24 

|«>13 

1^3 

bzz 

|*'34 

1^14 

1^24 

1^34 

644 

(7.85) 
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Then  the  estimated  stationary  points  for  the  AFM  fitted  response  surfaces  are 


(7.86) 


where  ©f  is  the  estimated  stationary  point  for  the  ECA^  response  surface  and  0^^^  is  the 
estimated  stationary  point  for  the  ACR^  response  surface.  For  EGA,  the  stationary  point  is 
approximately  9.11  coded  units  from  the  design  center,  which  is  far  outside  of  the  experimental 
design  region.  On  the  other  hand,  the  stationary  point  for  the  ACR  response  surface  is  only  0.84 
coded  units  from  the  design  center — well  within  the  design  region.  We  now  examine  the  eigenvalues 
of  B,  which  is  equal  to  one  half  of  the  Hessian  matrix,  to  determine  the  nature  of  the  stationary 
points.  The  eigenvalues  for  each  response  surface  are 


(7.87) 


indicating  that  both  stationary  points  are  saddle  points,  not  global  optima.  Hence  canonical  anal¬ 
ysis  or  ridge  analysis  is  required  to  identify  the  direction  for  further  experimentation,  if  required. 

We  use  ridge  analysis  to  identify  promising  points  for  further  investigation  of  the  experimental 
design  region.  Limited  ridge  analysis  results,  computed  by  the  SAS  statistical  computer  program, 
are  listed  in  Table  7.34.  The  table  includes  the  coded  distance  from  the  design  center,  estimated 
response,  standard  error  of  the  estimated  response,  and  aircraft  levels  for  4  selected  points  for  each 
performance  measure.  We  performed  AFM  replications  at  each  of  these  design  points  to  validate 
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the  ridge  analysis  and  have  provided  the  resulting  estimated  responses  at  those  points.  For  both 
measures,  improvements  in  the  levels  of  each  response  are  observed  until  the  last  point  replicated. 


Table  7.34  Ridge  analysis. 


ECA  Ridge  Analysis 

Coded 

Radius 

Estimated 

Response 

Standard 

Error 

Aircraft  ! 

lievel 

AFM 

Response 

CRAF 

C-141 

C-5 

C-17 

0.5 

53.88 

0.22 

15 

22 

45 

36 

53.00 

1.1 

58.60 

0.23 

14 

11 

48 

40 

56.53 

1.7 

63.31 

0.46 

13 

0 

51 

43 

58.64 

2.5 

70.22 

1.04 

12 

0 

55 

46 

56.72 

ACR  Ridge  Analysis 

1.0 

77.82 

2.62 

10 

14 

46 

38 

88.08 

1.6 

66.80 

4.84 

5 

7 

48 

41 

88.51 

2.3 

49.71 

10.39 

0 

0 

50 

44 

76.64 

2.0 

28.00 

18.35 

0 

0 

52 

47 

80.04 

The  AFM  ridge  anal3^is  results  indicate  that  we  just  missed  desigmng  a  second  order  design 
of  experiment  that  would  have  contained  the  feasibly  optimal  results.  In  particidar,  if  we  had  set 
the  low  levels  for  CRAF  and  C-141  aircraft  to  zero  instead  of  5,  the  '"best”  ridge  analysis  results 
would  have  been  in  the  design.  Overall,  the  results  obtained  with  our  second  order  design  are 
certainly  superior  compared  to  the  results  of  what  would  have  likely  been  the  second  order  design  if 
we  had  not  used  the  surrogate  search  method.  It  is  unlikely  that  an  analyst  would  have  set  CRAF 
and  C-141  aircraft  as  low  as  we  did  based  on  a  AFM  steepest  ascent  search.  Additionally,  for 
ECA  it  appears  that  there  is  actually  little  improvement  outside  the  actual  design  region  since  the 
mavirmiTn  observed  ECA  =  58.58  within  the  original  second  order  design  statistically  equivalent 
to  the  mayiTniim  observed  along  the  ridge  analysis  (58.64).  For  ACR,  it  appears  there  is  a  real 
improvement  to  the  observed  values  outside  of  the  second  order  design  region.  Of  course  these 
results  are  only  important  if  AMC  planner  are  interested  in  composing  air  lift  fleets  without  CRAF 
and/or  C-141  aircraft.  If  desired  additional  experiments  could  be  performed  to  further  define  the 
response  surfaces  to  include  these  new  points. 
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Tn  sununary,  we  have  demonstrated  the  effectiveness  of  the  surrogate  search  method  when 
applied  to  a  “real-world”  simulation  model  and  problem.  Despite  the  many  problems  in  applying 
the  method  to  AFM  and  this  particular  RSM  study,  namely  small  variance  for  the  performance 
measures  and  the  defining  of  appropriate  analytical  model  inputs  and  outputs,  the  surrogate  search 
method  was  quite  effective  in  defining  the  new  second  order  design  region.  Although  the  ACV 
method  was  not  efficient  in  reducing  study  times  at  the  initial  first  order  stage  of  the  study,  the 
additional  effort  in  validating  the  analytical  model  paid  off  in  the  end.  The  information  about 
the  predicted  behavior  of  AFM  gained  through  the  performance  of  numerous  surrogate  searches 
provided  us  with  a  second  order  design  region  that  nearly  encompassed  the  entire  range  of  “in¬ 
teresting”  AFM  output.  Without  that  additional  surrogate  search  knowledge  it  is  unlikely  that 
an  analyst  would  have  designed  a  set  of  experiments  that  covered  an  equivalent  region.  In  other 
words,  an  analyst  performing  the  same  RSM  study  without  a  surrogate  search  capability  would 
probably  establish  a  smaller  second  order  design  region  than  ours.  This  would  result  in  the  need 
for  another  gradient  search,  or  ridge  analysis  and  another  set  of  experiments  over  another  second 
order  design  to  reach  the  same  results  we  achieved  in  a  single  set  of  second  order  experiments. 
Hence,  we  conclude  that  the  surrogate  search  methodology  reduced  the  total  number  of  simulation 
replications  performed  by  a  factor  of  a  second  order  design  in  order  to  reach  satisfactory  results. 

7.^  Conclusion 

In  this  chapter,  we  have  demonstrated  the  effectiveness  of  the  surrogate  search  methodology 
for  two  different  simulation  models.  In  both  a  simple  study  and  a  complex  one,  we  have  shown 
that  the  surrogate  search  methodology  is  fully  integrated  within  the  context  of  a  simulation  RSM 
study.  It  provides  a  tool  to  the  analyst  to  not  only  reduce  study  times  but  to  also  enhance  the 
analyst’s  knowledge  of  the  system  imder  study.  Further,  the  flexibility  of  the  method  to  adapt  to 
different  situations  is  made  apparent  via  the  demonstrations.  For  simulation  models  that  are  used 
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on  a  regular  basis,  such  as  AFM,  the  cost  of  developing  and  validating  an  analytical  model  can  be 
repaid  as  the  surrogate  search  method  is  applied  in  several  different  studies. 
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VIIL  Summary  and  Recommendations 


8. 1  Overview 

This  dissertation  mates  significant  contributions  towards  the  synergistic  use  of  both  analytical 
and  simulation  models  to  reduce  the  time  required  to  complete  a  simulation  study.  The  particular 
advancements  in  the  field  of  using  both  types  of  models  in  concert  are  summarized  below,  followed 
by  suggestions  for  future  research. 

8.2  Contributions 

The  significant  contributions  achieved  by  this  research  are  summarized  below. 

8.2.1  A  CV  Monte  Carlo  Method.  As  mentioned  before,  previous  researchers  [48, 49, 53, 54] 
have  all  reported  unacceptable  levels  of  bias  when  using  ACV  controlled  estimators.  This  bias  is 
caused  by  the  necessity  to  evaluate  the  expected  value  of  the  ACV  given  the  distribution  of  the 
input  random  variables  used  to  produce  the  ACV  [49].  The  development  of  an  eflRcient  method 
to  resolve  the  bias  problem  using  a  general  Monte  Carlo  sampling  technique  makes  a  significant 
contribution  to  the  field  of  simulation  variance  reduction  and  the  synergistic  use  of  analytical  and 
simulation  models.  A  summary  of  this  work  has  been  accepted  for  publication  in  the  peer  reviewed, 
archival  journal,  HE  Transactions  [24]. 

8.2.2  ACV  Monte  Carlo  Method  with  Incomplete  Distributional  Knowledge.  The  ACV 
Monte  Carlo  method  described  above  relies  upon  the  knowledge  of  the  means  and  variances,  as 
well  as  the  approximate  distributions,  of  the  inputs  to  the  analytical  model.  In  this  dissertation, 
we  extend  the  range  of  permissible  simulation  models  where  the  ACV  method  can  be  applied  by 
describing  alternative  methods  of  generating  the  appropriate  random  vectors  used  to  approximate 
the  ACV  mean.  This  research  significantly  advances  the  field  of  ACV  application. 
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8.2.3  Surrogate  Search  Method.  A  new  method  for  performing,  and  justifying,  searches 
of  a  simulation  Hftsign  region  using  an  analytical  model  is  developed  in  this  dissertation.  The 
justification  for  the  method  is  adapted  from  classic  simulation  model  validation  and  verification. 
We  demonstrate  how  the  validation  and  surrogate  search  method  developed  in  this  research  is  fully 

r 

integrated  within  a  simulation  study  by  analyzing  the  results  of  the  ACV  method.  The  new  method 
and  successful  application  of  the  method  using  two  different  simulation  models  advances  the  field 
of  ACV’s  and  of  the  synergistic  use  of  both  analytical  and  simulation  models.  To  complete  this 
research  a  useful  analytical  model  of  the  Air  Force  simulation  model  MASS  is  developed  a  first 
in  the  field.  Additionally,  two  new  performance  measures  for  assessing  the  efficient  movement  of 
cargo  within  the  airlift  system  are  developed. 

8,3  Recommendations  for  Future  Research 

Related  topics,  which  could  not  be  completed  within  this  research,  are  described  below. 

The  first  research  topic  is  a  further  development  of  the  ACV  method  with  incomplete  distribu¬ 
tional  knowledge.  The  methods  described  in  this  research  could  be  tested  on  simulation  models  with 
more  variables  and/or  more  variance  in  the  output  statistics.  Further  research  into  non-parametric 
methods  of  distribution  sampling  could  also  be  explored. 

Another  area  of  possible  research  is  the  development  of  a  surrogate  screen  method.  In  this 
method,  the  goal  is  to  reduce  the  number  of  factors  in  a  simulation  experimental  design  by  again 
using  an  analytical  model  in  place  of  the  simulation  model.  A  proposed  method  could  be  an 
extension  of  the  surrogate  search  method  and  could  certainly  be  used  in  conjimction  with  the 
surrogate  search  method. 

More  research  into  the  application  of  the  surrogate  search  method  could  also  be  performed. 
For  example,  a  likely  area  of  fruitful  research  is  developing  methods  that  work  when  problems  occur 
when  attempting  to  apply  the  method.  We  touched  upon  some  of  the  difficulties  that  can  occur 
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when  appl3dng  the  method,  but  certainly  more  work  in  this  area  could  be  accomplished.  Also, 
research  into  further  uses  of  the  method  beyond  RSM  studies  could  certainly  be  performed. 

Finally  research  into  formulating  criteria  for  the  types  of  simulation  models  and  the  conditions 
that  are  necessary  for  successful  variance  reduction  using  the  ACV  method,  and  thus  successful  sur¬ 
rogate  searches,  could  be  performed.  Successful  completion  of  this  type  of  research  would  certainly 
be  beneficial  to  simulation  analysts. 
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Appendix  A.  Glossary  of  Acronyms  and  Abbreviations 

AB  Air  Base 
ACR  aircraft  cargo  ratio 
ACV  analytical  control  variate 
AFB  Air  Force  Base 
AMC  Air  Mobility  Command 
AFM  Airlift  Flow  Model 

BCMP  Baskett,  Chandy,  Muntz,  and  Palacios  (defines  a  type  of  queueing  network) 

BRACE  Base  Resource  and  Airfield  Capability  Evaluation 

CCD  central  composite  design 

CPU  central  processing  unit 

CRAF  Civil  Reserve  Air  Fleet 

CV  control  variate 

D  delay  station 

DOD  Department  of  Defense 

DOE  design  of  experiment 

ECA  early  cargo  per  aircraft 

ECV  external  control  variate 

FCFS  first  come,  first  served 

lAP  international  airport 

IID  independently  and  identically  distributed 

rVfeV  independent  validation  and  verification 
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JFK  John  F.  Kennedy 


LCFS  last  come,  first  served 
LOX  liquid  oxygen 

MASS  Mobility  Analysis  Support  System 

MOG  maximum  on  the  ground 

MSB  mean  square  error 

MSPE  mean  square  error  for  pure  error 

MVA  mean  value  analysis 

PC  personal  computer 

PS  processor  sharing 

RSM  response  surface  methodology 

SSE  sum  of  square  errors 

SSPE  sum  of  squares  for  pure  error 

SSPQ  sum  of  squares  for  pure  quadratic  terms 

SSR  sum  of  squares,  regression 

TPFDD  Time-Phased  Force  Deployment  Data  document 
USAF  United  States  Air  Force 
VRT  variance  reduction  tecbnique 
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